Beyond Language Bias: Overcoming Multimodal Shortcut and Distribution Biases for Robust Visual Question Answering
Jingliang Gu, Zhixin Li · 2024
Recent studies have found that many VQA models are influenced by biases, preventing them from effectively using multimodal information for reasoning. Consequently, these methods, which perform well on standard VQA datasets, exhibit underwhelming performance on the bias-sensitive VQA-CP dataset. Although numerous studies in the past have focused on mitigating biases in VQA models, most have only considered language bias. In this paper, we address the issue of bias in VQA task by targeting the various sources of bias. Specifically, to counteract shortcut biases, we integrate a bias detector capable of capturing both vision and language biases, and we reinforce its ability to capture biases using a generative adversarial network and knowledge distillation. To combat distribution bias, we use a cosine classifier to obtain a cosine feature branch from the base model, training it with an adaptive angular margin loss based on answer frequency and difficulty, along with a supervised contrastive loss to enhance the model's classification ability in the feature space. In the prediction stage, we fuse the cosine features with the prediction of the base model to obtain the final prediction of our model. Finally, extensive experiments demonstrate that our approach SD-VQA achieves state-of-the-art performance on the VQA-CPv2 dataset without using any data balancing, and achieves competitive results on the VQAv2 dataset.