NLP-based Traffic Scene Retrieval via Representation Learning
Touseef Sadiq, Christian Walter Peter Omlin · Proceedings of the World Congress on Electrical Engineering and Computer Systems and Science · 2023
Many automated systems require the interpretation of visual information, i.e., images, videos, and natural language input, i.e., speech or text, to comprehend their surroundings and communicate with interacting humans.One such hybrid application of computer vision using images and videos and natural language processing (NLP) recognizes traffic scenes, a crucial and challenging problem in automated transportation systems.Scene classification is just one of many areas where recent convolutional neural network (CNN) frameworks have proven to be highly effective.Still to be fully explored for application to problem-solving in the real world is CNN's impressive, truly representative learning capability.However, newer CNN implementations, such as YOLO and DeepSort, show promise for object detection.The BERT model is the benchmark for text embeddings and the most efficient method currently available.Hence, we aim to retrieve the vehicles from the traffic videos using natural language-based description, i.e., text.The paper proposes a novel approach that combines YOLOv7, the recent version of YOLO, DeepSort algorithms for object detection, i.e., detecting the vehicles from the traffic scene from the frames of the videos and the transfer learning model, i.e., BERT model for text embeddings.Additionally, a Kalman filter is utilized to track the cars by providing the id and will retain them in the other frames of the videos.The machine learning model performs the similarity checking, i.e., siamese neural networks.The experiments are performed on the standard dataset of AI city challenge 2022.Moreover, the results depict that the proposed approach achieves 28.49 % of Recall@5, 42.08 % of Recall@10, and 20.73 % of MRR, indicating the proposed method's effective approach.