ALDF: An Adaptive Logical Decision Framework for Multimodal Named Entity Recognition
Guohui Ding, Tianhao Jiang, Rui Hong Zhou, Qian Gao · 2024
Multimodal Named Entity Recognition (MNER) aims to achieve more accurate entity recognition by incorporating image information to assist text, which is particularly significant on social media platforms. Current research disproportionately emphasizes enhancing text with images, overlooking that the core of the NER task remains textual. The modal differences between images and text inevitably introduces noise when incorporating image information. Therefore, when textual information is sufficient to independently complete the NER task, the introduction of image information is unnecessary. This paper proposes an Adaptive Logical Decision Framework (ALDF) capable of determining the sufficiency of textual information in NER tasks, deciding whether to introduce image information, avoiding unnecessary noise, and focusing more on information-scarce entities when introducing image information. Specifically, we designed a Logic Reasoning Neural Network (LRNN) that uses an evidence-theory-based method to simulate human decision-making logic and generate decision support degrees for deciding whether image information should participate in the recognition task. When incorporating image information, we utilize the generated decision support degrees to guide the multi-head self-attention mechanism, enhancing the model's focus on information-scarce entities. Additionally, we employ a modality-aware progressive training method that can use decision information in real-time during multimodal training and reduce information redundancy between modalities. Extensive experiments demonstrate that our model achieves state-of-the-art performance on popular public datasets.