Towards Deconfounded Visual Question Answering via Dual-causal Intervention
Daowan Peng, Wei Wei · 2024
The Visual Question Answering (VQA) task has recently become notorious because models are prone to predicting well-educated "guesses" as answers rather than deriving them through visual understanding. The main culprit for this is that VQA models memorize the shortcut biases in the dataset during the training process. While a variety of solutions have been proposed, they solely focus on the shortcuts in the language modality, leaving other kinds of shortcut biases untouched. In this paper, we shift our lens to all kinds of shortcuts and resort to causal inference to circumvent these issues. Causal inference methods can discover the causal effect (P(Y|do(X))) [27] rather than statistic-based spurious correlations (P(Y|X)) in the dataset, making them naturally suitable for debiasing learning. To deconfound these shortcut biases, we propose a causality-aware method, coined as Dual Causal Intervention (DCI), to endow VQA models with better generalization by combining two components: linguistic backdoor intervention and visual front-door intervention. To be specific, we harness backdoor intervention to cut off the effects of confounders in the language modality and employ front-door intervention to eliminate the impact of confounders in the visual modality. We conducted extensive experiments on two challenging Out-of-Distribution (OOD) benchmarks, including VQA-VS and VQA-CE, which are designed to assess the robustness of VQA models under different shortcut biases. The experimental results show the effectiveness of our method. Specifically, our approach outperforms the current state-of-the-art debiasing methods on the IID metric and all nine OOD metrics of the VQA-VS dataset, and also surpasses the performance of the best-performing methods on all metrics of the VQA-CE dataset.