Efficient Hierarchical Multimodal Graph Neural Networks for AI-Driven Decision-Making in Consumer Data
Jothi Prakash V, S. Arul Antran Vijay, S. Gopikrishnan, Maha Driss, Wadii Boulila · IEEE Transactions on Consumer Electronics · 2025
The rapid increase in multimodal data (text, audio, and video) has enhanced consumer applications, such as sentiment analysis, emotion detection, and content retrieval. However, the efficient integration and alignment of such varied data streams remain a significant challenge. This paper introduces Hierarchical Multimodal Graph Neural Networks (HMGNN), a framework designed to optimize multimodal data processing by leveraging multi-scale graph structures. HMGNN captures both intra-and inter-modal relationships through Graph Neural Networks (GNNs), facilitating feature propagation across different modalities. The model incorporates self-supervised learning (SSL) and contrastive alignment strategies to enhance modality-invariant representations, improving the accuracy and efficiency of multimodal learning. To evaluate the performance of the proposed framework, several experiments were conducted and compared against 10 state-of-the-art models using three widely adopted datasets, namely, CMU-MOSI, CMU-MOSEI, and YouTube-8M. Experimental results demonstrate that our framework achieves an accuracy of up to 0.948 and a Multimodal Alignment Score (MAS) of 0.891 on the CMU-MOSEI dataset, consistently outperforming existing methods by 3–4%. Furthermore, HMGNN converges within 25 epochs, highlighting its computational efficiency. The proposed framework offers a scalable and effective solution for multimodal representation learning, making it well-suited for AI-driven decision-making in consumer electronics applications.