Feature Alignment and Reconstruction Constraints for Multimodal Sentiment Analysis
Qingmeng Zhu, Tianxing Lan, Jian Ping Liang, Deliang Xiang, Hao He · 2024
Sentiment is complex feedback of human perception of the outside world, which contains important potential information. Traditional sentiment analysis methods often use unimodal processing, which cannot accurately recognize complex emotional expressions. The existing multimodal sentiment analysis (MSA) methods based on feature fusion and post-fusion paradigms do not pay attention to the accuracy of the semantic expression of unimodal features and do not mine the correlation relationship between heterogeneous features, which leads to serious deviation of the fused multimodal sentiment. In this paper, a novel MSA method based on feature alignment and reconstruction constraints is proposed. The multimodal feature alignment module utilizes the cross-attention mechanism to establish the intrinsic connection between multimodal features and reduce the differences between multimodal heterogeneous features with the same sentiment. The multimodal feature reconstruction module is used to retain the unique semantics of unimodal features and reduce the loss of key information during heterogeneous feature alignment. Experimental results on the CH-SIMS v2.0 dataset show that the proposed method can significantly improve the model’s ability to cope with the recognition of complex sentiments.