Conditioned Masked Language and Image Modeling for Image-Text Dense Retrieval
Ziyang Luo, Yadong Xi, Rongsheng Zhang, Gongzheng Li, Zeng Zhao, Jing Ma · 2022
Image-text retrieval is a fundamental crossmodal task that takes image/text as a query to retrieve relevant data of another type.The large-scale two-stream pre-trained models like CLIP have achieved tremendous success in this area.They embed the images and texts into instance representations with two separate encoders, aligning them on the instance-level with contrastive learning.Beyond this, the following works adopt the fine-grained token-level interaction (Masked Language and Image Modeling) to boost performance further.However, the vanilla token-level objectives are not designed to aggregate the image-text alignment information into the instance representations, but the token representations, causing a gap between pre-training and application.To address this issue, we carefully design two novel conditioned token-level pre-training objectives, Conditioned Masked Language and Image Modeling (ConMLM and ConMIM), forcing models to aggregate the token-level alignment information into the instance representations.Combing with the instance-level contrastive learning, we propose our cross-modal dense retrieval framework, Conditioned Language-Image Pretraining (ConLIP).Experimental results on two popular cross-modal retrieval benchmarks (MSCOCO and Flickr30k) reveal the effectiveness of our methods.