Cross-lingual Cross-modal Pretraining for Multimodal Retrieval
Hongliang Fei, Tan Yu, Ping Li · 2021
Recent pretrained vision-language models have achieved impressive performance on cross-modal retrieval tasks in English.Their success, however, heavily depends on the availability of many annotated image-caption datasets for pretraining, where the texts are not necessarily in English.Although we can utilize machine translation (MT) tools to translate non-English text to English, the performance still largely relies on MT's quality and may suffer from high latency problems in realworld applications.This paper proposes a new approach to learn cross-lingual cross-modal representations for matching images and their relevant captions in multiple languages.We seamlessly combine cross-lingual pretraining objectives and cross-modal pretraining objectives in a unified framework to learn image and text in a joint embedding space from available English image-caption data, monolingual and parallel corpus.We show that our approach achieves SOTA performance in retrieval tasks on two multimodal multilingual image caption benchmarks: Multi30k with German captions and MSCOCO with Japanese captions.