Embedding Clustering using Lightweight Contrastive Learning For Cross-Modal Classification
José Amendola, Linga Reddy Cenkeramaddi, Ajit Jha · 2023
Deep learning models excel at computer vision tasks, but they require a lot of memory and computation power. Thermal image classification has gained popularity due to the robustness of this modality in the face of adverse weather and lighting conditions. Images from both visible and/or thermal modalities, on the other hand, must frequently be processed in memory constrained environments, such as embedded edge applications. We propose a novel cross-modal constrastive learning model with a novel objective function that learns and averages mappings from both visible and thermal modalities into a shared embedding space for cross-modal data clustering and classification. A lightweight architecture is used to generate embedding from images in both modalities, and the proposed algorithm yields cross-modal centroids that can be efficiently used for clustering and classification in either modality. We demonstrate the model's accuracy by running experiments on a dataset that contains both thermal and visible counterparts of the same objects.