Interpretable Multimodal Tucker Fusion Model With Information Filtering for Multimodal Sentiment Analysis
Xin Nie, Laurence Tianruo Yang, Zhe Li, Xianjun Deng, Fulan Fan, Zecan Yang · IEEE Transactions on Computational Social Systems · 2024
Multimodal sentiment analysis (MSA) integrates multiple sources of sentiment information for processing and has demonstrated superior performance compared to single-modal sentiment analysis, making it widely applicable in domains such as human–computer interaction and public opinion supervision. However, current MSA models heavily rely on black-box deep learning (DL) methods, which lack interpretability. Additionally, effectively integrating multimodal data, reducing noise and redundancy, as well as bridging the semantic gap between heterogeneous data remain challenging issues in multimodal DL. To address these challenges, we propose an interpretable multimodal Tucker fusion model with information filtering (IMTFMIF). We are the first to utilize the multimodal Tucker fusion model for MSA tasks. This approach maps multimodal data into a unified tensor space for fusion, effectively reducing modal heterogeneity and eliminating redundant information while maintaining interpretability. Furthermore, mutual information is employed to filter out task-irrelevant information and explain the association between input and output from an information flow perspective. We propose a novel approach to enhance the comprehension of multimodal data and optimize model performance in MSA tasks. Finally, extensive experiments conducted on three public multimodal datasets demonstrate that our proposed IMTFMIF achieves competitive performance compared to state-of-the-art methods.