Multimodal sentiment analysis using Multi-Layer Fusion Convolution Neural Network
D. Khalandar Basha, G Sunil, A. H. A. Hussein, Mukesh S, Myasar Mundher Adnan · 2023
The rapid development of social network helps the people to express the emotions and opinions through videos streaming in online platforms. But, most of the recent researches based on multimodal sentiment analysis does not perform an effective fusion on multimodal data. To overwhelm this issues, this research introduced a multilayer fusion Convolutional Neural Network (CNN) to perform a multimodal sentiment analysis. The data acquisition is performed from CMU-MOSI and CMU-MOSEI, the textual data is processed by tokenization and BERT; the video embedding is performed using FACET transformer and Mel-spectrogram with VGGish is used in encoding. At last, the decoding takes place using Multilayer Fusion CNN. The results obtained through experimental analysis shows that proposed approach achieved optimal accuracy of 94.61 % for MOSI dataset which is comparably higher than hybrid contrastive learning method, Multi-Tensor Fusion Network (MTFN), Sparse- and Cross-Attention Network (SCANET) and Adaptive Modality-specific Weight fusion network (AdaMoW) with accuracies of 85.2%, 80.9%, 85.98%, and 86.58% respectively.