Research on Transformer Multimodal Named Entity Recognition Based on Co-Occurrence Entities and Knowledge Enhancement
Panfei Liang, XU Hui-fang, Jiannan Xu, He Kejia · 2023
Transformers are important electrical equipment that plays an important role in the three stages of power generation, transmission, and distribution in the power system. Currently, research on transformers is mainly focused on single-modal fields such as images and audio, while research on multimodal named entity recognition is still in its infancy. This paper proposes a transformer multimodal named entity recognition model based on co-occurrence entities and knowledge enhancement to address the issues of image text mismatch and low efficiency of modal fusion in existing methods for multimodal named entity recognition (i.e. Co-occurrence Entities and Knowledge Enhancement, CEKE-MNER). The model adopts cooccurrence entities to solve the problems of image text mismatch and high noise and uses knowledge augmentation to reduce domain knowledge differences to solve the problem of entity semantic divergence. The experimental results on transformer data show that the CEKE-MNER model achieves an F1 value of 82.40% in multimodal data, which is about 2% higher than that in single-modal data; The experimental results on the publicly available datasets Twitter 15, Twitter 17, and Wu Kong CMNER show that the F1 values of the CEKE-MNER model are 72.76%, 85.49%, and 81.96%, respectively, which are higher than the current mainstream multimodal named entity recognition models and have good generalization performance.