Cross-Modal Relation and Sketch Prototype Learning for Zero-Shot Sketch-Based Image Retrieval
Yuanping Song, Yanwei Yu, Hao Tang, Junyuan Guo, Yibo Wang · 2022
Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) is an innovative cross-modal task that utilizes a sketch to retrieve corresponding images in the zero-shot learning scene. At present, most algorithms treat ZS-SBIR as a typical image classification problem, using image-level features with triplet or cross entropy loss to achieve retrieval, while ignore the correspondence between sketches and images on local features. Therefore, we propose a new Local Feature Contrastive Network (LFCN) for ZS-SBIR from the perspective of contrastive learning. More specifically, a local feature contrastive method is proposed to establish the cross-modal relationships between images and sketches with creatively applying transformers to extract similarity representation from sketch-image pairs to narrow the domain gap. Furthermore, a feature prototype memory bank is designed to learn sketch prototypes to address the in-class diversity problem in sketch domain. A large number of experiments show that our method notably superior to the state-of-the-art algorithms in both TU-Berlin and Sketchy datasets.