Two-stage Attention-based Fusion Neural Network for Image-Text Sentiment Classification
Xiaoran Hu, M. Yamamura · 2022
Social networks have gradually become an indispensable part of our daily lives, more and more people share their opinions and feelings through social software. As a result, Sentiment analysis, the core area of social media analysis, has been a hot research topic in recent years. Most previous work on multimodal sentiment analysis does not fully explore the information between text and image data and ignores the importance of analyzing fusion features. This paper presents a novel two-stage attention-based neural network to analyze textual-visual content for sentiment classification. The proposed network includes four sections: feature extraction, inter-modal feature learning, dual attention network, and sentiment classifier. The model first obtains multimodal features from pretrained models. Then in inter-modal feature learning, a cross-attention module and a self-attention module are adopted to correlate textual-visual features as well as explore independent individual information. Next, a dual attention network is proposed to obtain crucial representation from unimodal based bimodal features both in channel and position dimensions. Finally, a multilayer perceptron is applied to fuse multimodal information and predict sentiment deeply. The experimental results on two public multimodal sentiment datasets, MVSA-Single and MVSA-Multiple, show the effectiveness of the proposed model.