Multi-Modal Graph-Based Sentiment Analysis via Hybrid Contrastive Learning
Jiale Zhang · 2024
Multi-modal Sentiment Analysis (MSA) aims to analyze sentiments expressed across different modalities to gain deeper insights into emotional content. Previous researches have addressed numerous challenges in multi-modal sentiment analysis tasks, focusing on aspects such as incorporating multiple input modalities, handling complex edges, and utilizing advanced graph network architectures. However, there remains a deficiency in contrastive learning methods that accurately extract modal information, especially when using contrastive learning within and across modalities simultaneously. To tackle these issues, this paper proposes a multi-modal sentiment analysis framework named MGCP-CL based on hybrid contrastive learning. Specifically, a multi-modal graph is constructed through connecting nodes using both token-based and k-nearest strategies, followed by a graph pooling layer based on clustering techniques to preserve essential graph representations. Additionally, the proposed method integrates various forms of information derived from the network and utilize hybrid contrastive learning to enhance the generalization and robustness of representations, thereby facilitating the learning of intricate cross-modal relations. Extensive experiments demonstrate that our method achieves significant performance improvements compared to previous baselines on multi-modal sentiment analysis tasks.