Hierarchical Knowledge Stripping for Multimodal Sentiment Analysis
Aolin Xiong, Ying Zeng, Haifeng Hu · IEEE Transactions on Affective Computing · 2024
Multimodal sentiment analysis (MSA) has emerged as a prominent research area that focuses on leveraging multimodal data to understand intention and sentiment signals. Despite significant progress, two major challenges remain in integrating diverse modalities: modal heterogeneity and interference information. To address these issues, we propose a novel framework called Multimodal Hierarchical Knowledge Stripping (MHKS), which enables the progressive extraction of informative knowledge. First, inspired by the information bottleneck (IB), we design a hierarchical disentanglement strategy to stepwise separate task-relevant and task-irrelevant information at the feature, attribute, and semantic levels. This enables MHKS to extract valuable knowledge in unimodal representations and eliminate interference information. Then, to mitigate the distribution gap across multiple modalities, we further design an adaptive alignment strategy based on contrastive learning. We utilize text modality as a bridge to connect other nonverbal modalities, which encourages adaptive alignment across modalities and facilitates the learning of more harmonized joint representations. Comprehensive experiments on three popular datasets demonstrate our method achieves excellent performance on MSA tasks.