DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-training
Luyang Huang, Guocheng Niu, Jiachen Liu, Xinyan Xiao, Hua Wu · Findings of the Association for Computational Linguistics: ACL 2022 · 2022
Due to the limitations of the model structure and pre-training objectives, existing vision-andlanguage generation models cannot utilize pairwise images and text through bi-directional generation.In this paper, we propose DU-VLG, a framework which unifies vision-and-language generation as sequence generation problems.DU-VLG is trained with novel dual pre-training tasks: multi-modal denoising autoencoder tasks and modality translation tasks.To bridge the gap between image understanding and generation, we further design a novel commitment loss.We compare pre-training objectives on image captioning and text-to-image generation datasets.Results show that DU-VLG yields better performance than variants trained with uni-directional generation objectives or the variant without the commitment loss.On the image captioning task, our model reaches better performance than other pre-trained systems.On text-to-image generation datasets, our model achieves better or comparable results than previous state-of-the-art models.In addition, human judges further confirm that our model generates real and relevant images as well as faithful and informative captions.