Improving Zero-shot Adversarial Robustness via Integrating Image Features of Foundation Models
Koshiro Toishi, Keisuke Maeda, Ren Togo, Takahiro Ogawa, Miki Haseyama · 2024
The Contrastive Language-Image Pre-training (CLIP) model is one of the most successful multi-modal foundation models and is indispensable in the field of computer vision. While the CLIP model performs zero-shot image recognition tasks with high accuracy, it is vulnerable to adversarial examples created by white-box attacks against the CLIP model. To improve the zero-shot adversarial robustness of the CLIP model, almost all methods rely on adversarial training, and other approaches have not been sufficiently explored. In this paper, we propose a novel defense strategy that integrates the CLIP model with another foundation model, ImageBind, which has a different structure and learning mechanism from the CLIP model. Experimental results confirm the effectiveness of the proposed method, showing a significant improvement in the classification accuracy for adversarial examples targeting the CLIP model.