Self-Attentive CLIP Hashing for Unsupervised Cross-Modal Retrieval
Heng Yu, Shuyan Ding, Lunbo Li, Jiexin Wu · 2022
With the explosive growth of multi-modal data such as video, images, and text on the Internet, cross-modal retrieval has received extensive attention, especially the deep hashing method. Compared with the real-value method, deep hashing has shown promising prospects due to its low memory consumption and high searching efficiency. However, most existing studies have difficulties in effectively utilizing the raw image-text pairs to generate discriminative feature representations. Moreover, these methods ignore the latent relationship between different modalities and fail to construct a robust similarity matrix, resulting in suboptimal retrieval performance. In this paper, we focus on the unsupervised cross-modal hashing tasks and propose a Self Attentive CLIP Hashing (SACH) model. Specifically, we construct the feature extraction network by employing the pre-trained CLIP model, which has shown excellent performance in zero-shot tasks. Besides, to fully exploit the semantic relationships, an attention module is introduced to reduce the disturbance of redundant information and focus on important information. On this basis, we construct a semantic fusion similarity matrix that capable of preserving the original semantic relationships from different modalities. Extensive experiments show the superiority of SACH compared with recent state-of-the-art unsupervised hashing methods.