Fusion or Defusion? Flexible Vision-and-Language Pre-Training
Rongyi Sun, Ziran Li, Yifeng Ding, Qifan Wang, Jingang Wang, Hai-Tao Zheng, Wei Wu, Yunsen Xian · 2023
Existing approaches in the vision-and-language pre-training (VLP) paradigm mainly deploy either fusion-based encoders or dual-encoders, failing to achieve both effectiveness and efficiency in downstream multimodal tasks.In this paper, we build a flexible VLP model by incorporating cross-modal fusions into a dualencoder architecture, where the introduced fusion modules can be easily decoupled from the dual encoder so as to switch the model to a fusion-free one.To better absorb cross-modal features from the fusion modules, we design a cross-modal knowledge transfer strategy along with other comprehensive pre-training tasks to guide the training process, which can further strengthen both the fusion-based and fusionfree representation learning.Extensive experiments conducted on various downstream visionlanguage tasks show that our proposed model is well-equipped with effectiveness as well as efficiency, demonstrating a superior performance compared with other strong VLP models.