LaMHA: Efficient Multimodal Malicious Meme Classification via LoRA-Tuned Adaptation and Attentive-MLP Fusion
Meng Qi, Chung‐Lun Wei · 2025
To address insufficient cross-modal semantic alignment and limited dynamic discrimination in multi-modal malicious meme classification, this paper proposes a lightweight architecture integrating Low-Rank Adaptation (LoRA) and Attention-guided Multilayer Perceptron (aMLP). The architecture achieves efficient and accurate malicious meme classification through a two-stage cascade: 1) The LoRA module embedded in CLIP's text encoder performs lightweight text feature projection via low-rank matrix decomposition, enabling efficient cross-modal adaptation with only 0.3% fine-tuned parameters; 2) The aMLP module at the classification layer dynamically fuses multi-modal features using cross-modal attention weights and suppresses noise via a gating mechanism. Experiments on HMC and HarMeme datasets demonstrate superior performance in Accuracy (77.74% / 83.72%), F1 (69.79% / 79.82%), and AUROC (84.01% / 92.16%), with a notable 4.33% AUROC gain on cultural metaphors. Ablation studies confirm nonlinear synergistic effects (11.53% F1 improvement on HMC). This work provides an efficiency-precision balanced solution for multi-modal content safety.