Multimodal Deep Learning for Online Meme Classification
Stephanie Han, Sebastian Leal-Arenas, Eftim Zdravevski, Charles C. Cavalcante, Zois Boukouvalas, Roberto Corizzo · 2024
Memes possess a humorous intent, yet they can also be used for malicious purposes. Analysing meme data has the potential to enhance content monitoring, identify emerging topics, and support content moderation in online platforms. Memes also represent an interesting use case for multimodal machine learning, as they combine text and image data. In this study, we explored the linguistic characteristics and analysed the convergent themes of five meme classes through common word extraction. Moreover, we compared the effectiveness of various machine learning models, i.e., unimodal (text or image) and multimodal (early fusion, late fusion) in binary and multiclass meme classification tasks. Our results on a large meme dataset showed that memes heavily adhered to current affairs, demonstrated by the high frequency of topical words across meme classes. Regarding model accuracy, early fusion achieved superior accuracy over late fusion in meme classification. Binary models outperformed multi-class classification methods. However, fusion models did not consistently surpass the accuracy of independent text or image-based models.