MTCA: A Multimodal Summarization Model Based on Two-Stream Cross Attention
Qiduo Lu, Ye Xia, Chenhao Zhu · 2022 2nd International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI) · 2022
The combination of vision and natural language patterns has become an important topic in the field of computer vision and natural language processing. With the rapid growth of multimodal information, multimodal summarization has received unprecedented attention. In this paper, we propose MTCA which composed of a pre-trained feature extractor, a text encoder, an image encoder, a two-stream cross attention fusion module and a summary decoder to complete the multimodal summarization task. This framework integrates three sub tasks: extractive summarization, abstractive summarization and image selection. At the same time, we introduce a residual network to alleviate the problem of modality-bias. Experiments show that the model is better than the baseline model and the performance is better than the text summarization method which ignores visual modality.