Multimodal Product Matching and Category Mapping: Text+Image based Deep Neural Network
Ketki Gupte, Linsey Xiaolin Pang, Harshada Vuyyuri, Sujitha Pasumarty · 2021 IEEE International Conference on Big Data (Big Data) · 2021
In the expanding world of online retail, there exists an extensive catalog of products in the current retail markets. Different retailers and e-commerce sites have millions of product images and text descriptions. Matching the products across the universe proves to be an important and challenging task to determine if specific products exist in our catalog. Expanding upon our previous work on product matching and category mapping from textual descriptions using transformer based models, we propose a weighted multi-modal approach for product matching by using both images and text within the training and matching process. We integrate both transformer and ResNet architectures into the siamese network to generate fine-tuned product embedding. Extensive experiments are conducted to evaluate our proposed weighted multi-modal approach comparing with single-modal approaches. The experiments show our proposed approach outperforms single-modal approaches.