Ontology-Semantic Alignment On Contrastive Video-Language Model for Multimodel Video Retrieval Task
Yifan Zhou, Yizhou Ding, Yuwu Dong, Hao He · 2023
Contrastive Learning-based models have shown impressive performance in text-image retrieval tasks. However, when applied in video retrieval, traditional contrastive learning strategies have faced challenges in achieving satisfactory results due to redundancy of video contents. We discern several potential reasons: (1)Current methodologies sometimes overlook the significant information imbalance between videos and query text, specifically neglecting the in-depth textual representation of the content within the videos. (2) Current video matching methodologies typically focus on cross-model alignment at general entity similarity level, without specific consideration for how entity pair preferences and similarity properties affect the task at hand. (3) Previous vectorized retrieval based on video content features have been somewhat flawed. They primarily focused on aligning overall features without having an video content tags feature for meaningful feature discrimination. Considering the shortcomings identified in the mentioned three aspects, we propose an ontology semantic labels augments retrieval model and introduce a method to integrate video ontology semantic labels into the contrastive learning framework. In particular, we have developed ontology semantic descriptions about entities encompassing both human figures and textual elements within the videos. Subsequently, we conducted training and testing on the CMIVQA dataset to assess the performance of our approach. The experimental results show that employing fine-grained ontology labels as sample pairs for contrastive learning leads to an increased level of precision in video retrieval tasks.