Enhancing Multimodal Tweet Analysis Accuracy through Integration of CLIP Model and Multi-layer Attention Mechanism
Yupu Liu, Kazuyuki Matsumoto · 2024
This study employs a multimodal approach combining the CLIP model with multi-layer attention mechanisms to analyze sentiments and identify trending content in tweets.We utilize ViT and BERT to extract image and text features, respectively, and then re-extract these features using the CLIP model.Attention layers are introduced to enhance the correlation between the extracted features, thereby improving the model's accuracy.Experimental results show that our model achieves an accuracy of 67.6% on an English dataset and 81.0% on a Japanese dataset.By using the proposed analysis model to analyze text and image information on Twitter, we can accurately capture sentiment trends and popularity metrics, aiding applications such as social media trend analysis, influencer impact assessment, and market trend prediction.