UCMIB-PNS: Balancing Sufficiency and Necessity With Probabilistic Causality and Cross-Modal Uncertainty in Multimodal Sentiment Analysis
Jili Chen, Yihua Zhong, Qionghao Huang, Changqin Huang, Fan Jiang, Xiaodi Huang, Xun Wang · IEEE Transactions on Affective Computing · 2025
Multimodal sentiment analysis aims to accurately identify sentiment orientations by integrating information from multiple modalities such as text, audio, and video. However, a key challenge in multimodal fusion is effectively balancing the sufficiency and necessity of information across modalities. Traditional models often fail to qualify and capture this balance due to the presence of noise and redundant information in multimodal data, leading to suboptimal performance in sentiment analysis. To address this issue, we propose a novel multimodal sentiment analysis method calledUCMIB-PNS, which is guided by information bottleneck and probabilistic causality. The method employs anUncertainCross-ModalInformationBottleneck(UCMIB)module to reduce redundant information within modalities and maximize discriminative information. The UCMIB utilizes codebooks to dynamically record the distributions of samples and employs random sampling to conduct uncertain modeling across different modalities. It integrates uncertainty-aware contrastive learning and KL divergence for dynamic comparison and compression of information from different modalities. Moreover, UCMIB-PNS uses differentiableProbability ofNecessity andSufficiency(PNS)estimators to estimate and re-weight the sufficiency and necessity of modalities by constructing several counterfactual scenarios through end-to-end learning. Experiments conducted on four publicly available multimodal sentiment analysis datasets demonstrate that UCMIB-PNS achieves optimal performance on both clean and noisy data. Extended experiments further validate the method's robustness under different types of noise.