ConFEDE: Contrastive Feature Decomposition for Multimodal Sentiment Analysis
Jiuding Yang, Yakun Yu, Di Tao Niu, Weidong Guo, Yu Xu · 2023
Multimodal Sentiment Analysis aims to predict the sentiment of video content.Recent research suggests that multimodal sentiment analysis critically depends on learning a good representation of multimodal information, which should contain both modality-invariant representations that are consistent across modalities as well as modality-specific representations.In this paper, we propose ConFEDE, a unified learning framework that jointly performs contrastive representation learning and contrastive feature decomposition to enhance representation of multimodal information.It decomposes each of the three modalities of a video sample, including text, video frames, and audio, into a similarity feature and a dissimilarity feature, which are learned by a contrastive relation centered around text.We conducted extensive experiments on CH-SIMS, MOSI and MOSEI to evaluate various state-of-the-art multimodal sentiment analysis methods.Experimental results show that ConFEDE outperforms all baselines on these datasets on a range of metrics.