Deep Image Annotation and Classification by Fusing Multi-Modal Semantic Topics
Yongheng Chen, Fuquan Zhang, Wanli Zuo · KSII Transactions on Internet and Information Systems · 2018
Due to the semantic gap problem across different modalities, automatically retrieval from multimedia information still faces a main challenge.It is desirable to provide an effective joint model to bridge the gap and organize the relationships between them.In this work, we develop a deep image annotation and classification by fusing multi-modal semantic topics (DAC_mmst) model, which has the capacity for finding visual and non-visual topics by jointly modeling the image and loosely related text for deep image annotation while simultaneously learning and predicting the class label.More specifically, DAC_mmst depends on a non-parametric Bayesian model for estimating the best number of visual topics that can perfectly explain the image.To evaluate the effectiveness of our proposed algorithm, we collect a real-world dataset to conduct various experiments.The experimental results show our proposed DAC_mmst performs favorably in perplexity, image annotation and classification accuracy, comparing to several state-of-the-art methods.