Transformer based multimodal similarity search method for E-Commerce platforms
Chandan Charchit Sahoo, Deepak Singh Tomar, Jyoti Bharti · 2023
Multimodal similarity search has been attracting much attention from scholars in recent times. The intensification of research into Multimodal search lies in the motive to understand the context of the multiple modes associated with the data and the interdependency of these modes. Multimodal data can be in the form of audio, video, images, texts, and any combination of them. For relevant extraction of features from them can take too much time and thus it may cause the whole process of learning the features and searching for them time-consuming. In this work, a deep neural network uses Convolutional neural network and Bidirectional Encoder Representations from Transformers to learn features from text and image data. A K nearest neighboralong with KD Tree has been used to search for similar image-text pairs. Evaluations based on several image-text multimodal retrieval procedures show the effectiveness of the proposed multimodal search method and its higher speed performance compared to existing algorithms.