TNOD: Transformer Network with Object Detection for Tag Recommendation
Kai Feng, Tao Liu, Heng Zhang, Zihao Meng, Zemin Miao · 2023
The hashtag is an effective tool to manage and distribute social media content in recent years. Most existing tag recommendation methods rely on user profiles to improve the F1 score by roughly 20%. In the final results, multimodal information accounts for 58% of the total, while user information accounts for 42%. However, these methods neither provide a sufficient fusion method of information across modalities nor ignore the visual information from the main sources of tags. In this paper, we propose a novel model entitled Transformer Network with Object Detection (TNOD), which utilizes the contextual semantics of entities combined with text and image information, forming a multi-layer attention mechanism. In particular, we use object detection to extract entities in images and fuse the relationship between entities, text, and images with a multi-layer attention mechanism. Experiment results validate the superiorities of our proposed scheme from the perspective of recall rate and precision rate.