Multimodal Hateful Meme Detection with Graph Attention Networks and Contextual Cues
Hunjun Shin, Dhruv Agarwal, Wonhee Lee, Mahdi Imani, Naveen Naik Sapavath · 2025
The classification of hateful memes remains a challenging task due to their multimodal nature, where the interplay of textual and visual elements often conveys implicit or nuanced harmful content. This paper introduces a novel classification framework leveraging Graph Attention Networks (GATs) to model cross-modal relationships between visual and textual components. The proposed method integrates Visual Question Answering (VQA) and image captioning to enhance contextual understanding and refine semantic representations of multimodal data. Each meme is represented as a fully connected graph, where nodes correspond to embeddings derived from visual features, captions, and VQA responses, while GATs dynamically assign importance to these relationships. Experimental evaluation on the HarMeme dataset demonstrates the effectiveness of our approach, achieving competitive accuracy and AUROC compared to unimodal baselines (ResNet, DistilBERT) and showing promising improvements over existing multimodal models such as Contrastive Language Image Pre-training (CLIP). These results highlight the potential of the proposed GAT-based architectures for improving hateful meme detection and advancing multimodal content analysis.