A Novel Dual-Pipeline based Attention Mechanism for Multimodal Social Sentiment Analysis
Ali Braytee, Andy Shueh-Chih Yang, Ali Anaissi, Kunal Chaturvedi, Mukesh Prasad · 2024
Traditionally, sentiment analysis methods rely solely on text or image data. However, most user-generated social media content includes both textual and image content. In this study, we propose a novel Dual-Pipeline based Attentional method that uses different modalities of data, including text and images, to analyse and interpret emotions and sentiments expressed in tweets. Our proposed method simultaneously extracts meaningful local and global contextual features from multiple modalities. Local fusion layers within each pipeline combine modality-specific features using an attention mechanism to enrich the joint multimodal representation. A global fusion layer consolidates the collective sentiment representation by seamlessly intermixing the outputs of both pipelines. We evaluate our proposed method using performance metrics such as accuracy and F1-score. Through extensive experimentation on the MVSA dataset, our method demonstrates superior performance compared to state-of-the-art techniques in identifying the sentiment conveyed in social media data.