Multimodal sarcasm semantic detection based on inter-modality incongruity
Zhangmingjia Yin, Fucheng You · International Conference on Computer Graphics, Artificial Intelligence, and Data Processing (ICCAID 2021) · 2022
The irony is a complex linguistic phenomenon used to express the opposite of its true meaning and is a more difficult part of sentiment analysis. Multimodal sentiment analysis complements each other with multiple modal data, thus reflecting the user's sentiment tendency more accurately. Sarcasm is often caused by semantic inconsistencies between modalities. In this paper, we focus on multimodal sarcasm semantic detection by analyzing the intra- and inter-modal inconsistencies. Firstly, image features, image attributes are extracted using ResNet, and then text features and label features are extracted using a pre-trained BERT model. The image features with image attribute splicing and text features are input into the self-attentive mechanism, and output inter-modal inconsistency. The text features and label features are input into the common attention mechanism and the output intra-modal inconsistency. After experiments sarcasm, semantic prediction accuracy can reach 86%.