BCD-MM: Multimodal Sentiment Analysis Model With Dual-Bias-Aware Feature Learning and Attention Mechanisms
Lei Ma, Jingtao Li, Dangguo Shao, Jiangkai Yan, Jiawei Wang, Yukun Yan · IEEE Access · 2024
Multimodal Sentiment Analysis (MSA) is gaining attention. But faces two main challenges: effective feature extraction across modalities without redundancy, and removing spurious correlations between sentiment labels and multimodal features. In this paper, we propose a novel multimodal learning debiasing model, named Bilateral Cross-modal Debias Multimodal sentiment analysis Model (BCD-MM), to address these issues. To tackle the first challenge, our model uses an attention score-based method for critical information preservation and redundancy elimination within modalities, alongside a gated cross-modal attention mechanism for filtering inconsistencies through modal interaction. For the second challenge, BCD-MM features a debiasing approach with double bias extraction, utilizing a Tanh-based Mean Absolute Error (TMAE) loss function and inverse probability weighting to mitigate spurious correlations. The main objective of our model is to enhance the model’s ability to generalize in out-of-distribution (OOD) situations by reducing the reliance on non-causal correlations. Extensive testing on three public datasets (MOSI, MOSEI, and SIMS) and two OOD datasets (OOD MOSI and OOD MOSEI) demonstrates our model’s effectiveness in MSA and debiasing tasks.