A Multimodal Named Entity Recognition Method with Dynamic Bidirectional Gating Mechanism
Hao Chen · 2023
A multi-modal named entity recognition (MNER) method based on a dynamic bidirectional gating mechanism (DBiGM) was proposed to solve the ambiguity problem of traditional named entity recognition in text data due to the need for more context information and the inability to integrate multi-modal data information. By using a pre-trained BERT network model and an improved ResNet101 network for data fusion of text images, the Dynamic Gating Mechanism (DGM) receives a hidden state as input and outputs a scalar value as a gating factor. The attention mechanism is used to generate a new context vector on the sequence output and merge it with the original sentence sequence output, thus allowing the model to adaptively select the appropriate input sequence. The experimental consequences demonstrate that compared to the baseline method, the presented multimodal named entity recognition model with DBiGM has improved the F1, precision, and recall on the public datasets Twitter2015 and Twitter2017 to some extent.