AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
Zhongzhi Chen, Guang Liu, Bowen Zhang, Qinghong Yang, Ledell Wu · 2023
CLIP (Contrastive Language-Image Pretraining) is an English multimodal representation model learned from a massive amount of English text-image pairs and has achieved great success in various downstream tasks, including image classification, text-to-image retrieval, and image generation.When extending CLIP to other languages, the major problem is the lack of good-quality text-image pairs.In this work, we present AltCLIP, a simple and lowresource method to build a strong multilingual multimodal representation model.Instead of training a model from scratch on multilingual text-image pairs, we take the original CLIP model trained on English text-image pairs and alter its text encoder with a pre-trained multilingual text encoder (XLM-R).We then align text and image representations by a two-stage training schema consisting of teacher learning and contrastive learning.Our method utilizes the existence of rich parallel text data and pre-trained multilingual language models.We present extensive experimental evaluations to demonstrate the effectiveness of our proposed method.Our model sets new state-ofthe-art zero-shot performances on a wide range of tasks in multilingual multimodal benchmarks, including ImageNet-CN/IT/JA/KO serials, Flicker30k-CN, COCO-CN, Multi30k, and XTD.Further, our model outperforms the original CLIP model on zero-shot crossmodal retrieval, Image Classification in the Wild (ICinW) tasks, and CLIP Benchmark.We open-source our code, pre-trained model weights, and evaluation toolkit of multilingual multimodal tasks, to facilitate research on multilingual multimodal representation learning.