InterBERT: An Effective Multi-Modal Pretraining Approach via Vision-and-Language Interaction
Junyang Lin, Yang An, Yichang Zhang, Jie Liu, Jingren Zhou, Hongxia Yang · arXiv (Cornell University) · 2020
We propose a novel method for multi-modal pretraining, namely InterBERT (BERT for Interaction). The proposed architecture owns a strong capability of modeling interaction between the information flows of different modalities. The single-stream interaction module is capable of effectively processing information of multiple modalities, and the two-stream extraction module on top preserves the independence of each modality to avoid significant performance downgrade in single-modal tasks. The proposed pretraining task called masked group modeling (MGM) includes masked segment modeling and masked region modeling. It encourages the model to model a span or region instead of a single word or object, and it requires the model to learn from the general context. We pretrain the model with MGM and the conventional image-text matching, and finetune it on a series of vision-and-language downstream tasks, including caption-based image retrieval, zero-shot image retrieval, and visual commonsense reasoning. Experimental results demonstrate that InterBERT outperforms a series of strong baselines, including the most recent multi-modal pretraining methods. The analysis shows that the proposed MGM is effective for pretraining, and our method for multi-modal pretraining can adapt to single-modal tasks without significant performance decrease in comparison with the BERT-base model.