D2TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization
Yunlong Liang, Fandong Meng, Jiaan Wang, Jinan Xu, Yufeng Chen, Jie Zhou · 2023
Many-to-many multimodal summarization (M 3 S) task aims to generate summaries in any language with document inputs in any language and the corresponding image sequence, which essentially comprises multimodal monolingual summarization (MMS) and multimodal cross-lingual summarization (MXLS) tasks.Although much work has been devoted to either MMS or MXLS and has obtained increasing attention in recent years, little research pays attention to the M 3 S task.Besides, existing studies mainly focus on 1) utilizing MMS to enhance MXLS via knowledge distillation without considering the performance of MMS or 2) improving MMS models by filtering summaryunrelated visual features with implicit learning or explicitly complex training objectives.In this paper, we first introduce a general and practical task, i.e., M 3 S. Further, we propose a dual knowledge distillation and target-oriented vision modeling framework for the M 3 S task.Specifically, the dual knowledge distillation method guarantees that the knowledge of MMS and MXLS can be transferred to each other and thus mutually prompt both of them.To offer target-oriented visual features, a simple yet effective target-oriented contrastive objective is designed and responsible for discarding needless visual information.Extensive experiments on the many-to-many setting show the effectiveness of the proposed approach.Additionally, we will contribute a many-to-many multimodal summarization (M 3 Sum) dataset. 1