Multimodal Learning with Incompleteness towards Multimodal Sentiment Analysis and Emotion Recognition Task
Cam-Van Thi Nguyen, Huy-Sang Nguyen, Duc-Trong Le, Quang-Thuy Ha · 2023
Multimodal machine learning tasks have gained significant popularity and demonstrated promising results in mul-timodal data analysis. However, existing multimodal approaches primarily focus on scenarios where all modalities have complete data, neglecting the challenges posed by missing data. In real-world settings, certain modalities may lack parallel sequences due to noise during data collection and preprocessing, leading to random missingness in each modality. In this paper, we propose an integrated model MM-Align+ that leverages information from three input modalities: audio, visual, and text, to address two important multimodal tasks: sentiment analysis and emotion recognition under missing data with varying rates. We conduct extensive experiments on three public datasets, namely MOSEI, MOSI, and MELD. The preliminary results demonstrate the promising performance of the MM-Align+ model and suggest potential for further improvements and novel insights.