Visual AI for Satellite Imagery Perspective: A Visual Question Answering Framework in the Geospatial Domain
Siddharth Bhorge, Milind Rane, Nikhil Rane, Mandar Patil, Prathamesh Saraf, Jyotika Nilgar · 2023
This study presents a task of visual question answering (VQA) for satellite imaging datasets. Satellite images provide a large-scale dataset that can be used for various purposes such as land cover classification, item counting, and detection. This project investigates the use of human intent questions for effective method for obtaining data from satellite imaging data. Users can query the system to retrieve meaningful data about regarding image entities or relationships among items depicted in the images. Developing high-performing and reliable VQA systems is a challenging task, particularly in the context of disaster management and infrastructure. In this work, we examine traditional combination methods such as addition, concatenation, and feature-wise multiplication using state-of-the-art image and text feature extraction models. We present simple and efficient system that outperforms existing techniques on the FloodNet dataset and achieves state-of-the-art performance. Our streamlined solution requires significantly less training and inference time than traditional VQA systems. We also evaluate the performance of several backbones and present their aggregated results. We propose to use a visual model to extract context information, translate it to text, and inject it into a language model. The language model then processes the question in conjunction with the visual context and extracts the information needed to generate a response.