Stacked multi-modal refining and fusion network for visual entailment
Yuan Yao, Min Qiang Hu, Xiaohua Wang, Chuqing Liu · Thirteenth International Conference on Graphics and Image Processing (ICGIP 2021) · 2022
Recently, Visual Entailment is proposed as a new task in the multi-modal field. Its main focus is to reason about entailment relations between a real-world image as a premise and a natural language as a hypothesis. Some papers have proposed models to obtain more accurate entailment relation judgments. However, these models do not consider the semantics of text at both global and local granularity, and the refining of the two modalities is not sufficient. In this paper, a new stacked multi-modal refining and fusion network is proposed. For cross-sniffing and key information activation between global and local features of hypothesis-sentence, a Global & Local Textual Features Fusion block is introduced. Secondly, a Refining and Affine Fusion block is proposed to achieve efficient multi-modal attention and fusion between image and text features. Finally, this paper presents a stacked structured network which embedded an adaptive hypothesis-preserving mechanism to enriching the grounds for semantic implication judgements. The experiments demonstrate that our model has a certain improvement in the accuracy of visual entailment classification compared with some existing methods in this field.