Joint Adversarial Purification: Mitigating the Threat of Multimodal Adversarial Examples
Qin Li, Youze Wang, Wenbo Hu, Richang Hong · 2025
Vision-language pre-training (VLP) models exhibit exceptional generalization capabilities across diverse vision-language (V+L) tasks. However, studies reveal their vulnerability to carefully crafted multimodal adversarial examples. Notably, state-of-the-art attacks like Co-Attack (white-box) and SGA (transfer-based) demonstrate alarming success rates, posing critical security threats to VLP models. Current defense mechanisms against such multimodal attacks remain insufficiently explored. To address this challenge, we propose Joint Adversarial Purification (JAP), a novel defense framework that synergistically eliminates adversarial perturbations across modalities through cross-modal interaction. Our approach harnesses cross-modal semantic synergy to jointly purify adversarial perturbations: Generative denoising establishes visual-semantic anchors through diffusion processes, while purified linguistic cues conversely enhance visual perturbation filtering, forming a self-reinforcing defense cycle. Extensive experiments demonstrate that JAP effectively mitigates adversarial threats from both white-box Co-Attack and transfer-based SGA, significantly outperforming existing unimodal defense baselines. This work establishes a new paradigm for securing VLP models against multimodal adversarial attacks.