SAMNER: Image Screening and Cross-Modal Alignment Networks for Multimodal Named Entity Recognition
Luyi Yang · 2024
With the proliferation of social media data, Multimodal Named Entity Recognition (MNER) has received much attention; using different data modalities is crucial for the development of natural language processing and neural networks. However, existing methods suffer from two drawbacks: 1) text-image pairs in the data only sometimes correspond to each other, and it is impossible to rely on contextual information due to the short text nature of social media. 2) Despite the introduction of visual information, heterogeneity gaps may occur in previous complex fusion methods, leading to misidentification. This paper proposes a new synthetic image with a selected graphic alignment network(SAMNER) to address these challenges and construct a matching relationship between external images and text. To solve the graphic mismatch problem, we use a stable diffusion model to generate the images and perform entity labeling. Specifically, we generate images and perform entity labeling through the stable diffusion model to generate the image with the highest match to the text, filter the generated images by the internal image set to generate the best image, and then perform multimodal fusion to predict the entity labeling, we design a simple and effective multimodal attentional alignment mechanism to obtain a better visual representation, and we conduct a large number of experiments. The experiments prove that our model produces competitive results on the two publicly available datasets.