Multimodal rough set transformer for sentiment analysis and emotion recognition
Xinwei Sun, Huijie He, Haoyang Tang, Kai Zeng, Tao Shen · 2023
Sentiment analysis and emotion recognition are crucial tasks that utilize multimodal information. Transformer models have shown exceptional performance in multimodal fusion. However, traditional dot product transformers do not tolerate uncertainty inside sentiment analysis and emotion recognition data. In this study, we introduce rough set self-attention and rough set cross-attention mechanisms for multimodal sentiment analysis and emotion recognition. A common concept is established based on granulation relations to extract important features through approximation. We then investigate a multimodal fusion transformer network based on rough set theory, which facilitates the interaction of multimodal information and feature guidance through rough set cross-attention. Our empirical findings demonstrate that this is the first integration of rough set theory and transformer mechanisms for multimodal sentiment analysis and emotion recognition. Compared to traditional transformer fusion methods, our model can handle uncertain information and provides stronger global relationship guidance, allowing for better extraction of semantic information from multimodal data. We evaluate our model on sentiment analysis and emotion recognition experiments using the CMU-MOSEI and MELD datasets. The results show that our method outperforms state-of-the-art networks.