NewsInsight: A Comprehensive Video Event Retrieval System with Spatial Insights and Query Assistance
Gia-Huy Vuong, Van-Son Ho, Tien-Thanh Nguyen-Dang, Xuan-Dang Thai, Van-Tu Ninh, Minh-Khoi Pham, Tu-Khiem Le, Graham F. Healy, Minh–Triet Tran · 2023
Video event retrieval is the task of finding videos that are relevant to a given query. It is a challenging problem because videos are typically much larger than images, and they can contain a variety of different objects and scenes. However, there are a number of different approaches to video retrieval, and the field is rapidly evolving. Some of the most promising research directions include the use of deep learning and multimodal features. In this paper, we introduce NewsInsight – a comprehensive video event retrieval system developed for participating AI Challenge 2023. The system under investigation leverages the Bootstrapping Language-Image Pre-training (BLIP) model for zero-shot image-text retrieval, demonstrating superior recall scores on the Flickr30K dataset compared to the Contrastive Language–Image Pretraining (CLIP) model. In addition, it employs an Elastic Search filtering mechanism to discard irrelevant images. Beyond semantic search mechanisms, the system supports visual similarity search by calculating the inner product distance between vectors in the video frames corpus and the query image. The system also incorporates an explicit relevance feedback function, AI-based query description rewriting, and visual-example-generating features, enhancing the precision of the query description and aiding end-users in formulating a more accurate depiction of the targeted image for retrieval.