Multimodal Knowledge Reasoning for Enhanced Visual Question Answering
Afzaal Hussain, Ifrah Maqsood, Muhammad Khuram Shahzad, Muhammad Moazam Fraz · 2022
In the domains of Natural Language Processing (NLP) and Computer Vision (CV) Visual Question Answering (VQA) is a multidisciplinary task, in which an image and a question are given to a VQA system, which is responsible for giving the answer. The VQA system is used for a variety of real-world applications, such as providing situational information based on visual material, making judgments using a vast quantity of surveillance data, interacting with robots, and helping individuals who are blind or visually impaired. Although it is required yet challenging to complete comprehensive VQA, Fact-based VQA (FVQA) approaches in which external knowledge is required to process with image and question. Existing FVQA methods combine all types of data without fine-grained selection, thereby generating unexpected noise while reasoning about the final result. The problem solution should be able to collect complementary-information evidence based on question-attention. We represent an image with different layers of information by a multimodal knowledge graph relating to the features of visual, factual, and semantic. We propose a multimodal knowledge graph-convolutional-network (GCN) to collect relevant-information evidence from different information layers based on the given question. In particular, intra-modal knowledge graph attention takes evidence from each modality, while inter-modal knowledge graph attention gets evidence across the different information layers. To get an optimal answer, we stack this process multiple times to perform a reasoning mechanism. Over the FVQA dataset, we achieved state-of-the-art results by improving 10.86% test accuracy, which demonstrates the usefulness and interpretability of our approach.