SSTAP: Generating Sample-Specific Transferable Adversarial Patch in Multimodal Contrastive Learning
Changchun Yin, Liming Fang · 2025
The growing use of multimodal contrastive learning in critical applications demands robustness against adversarial attacks. Although universal adversarial patches can broadly impact downstream tasks, their fixed perturbations are easily detectable and can be mitigated by simple defenses. To address this, we propose the sample-specific transferable adversarial patch (SSTAP), which generates adversarial patches tailored to individual inputs. By exploiting the unique features of each sample, SSTAP creates imperceptible patches that disrupt feature representations across diverse downstream tasks. Experiments on Wikipedia and Pascal-Sentence datasets show significant performance drops, demonstrating SSTAP's effectiveness.