Generalization Differences between End-to-End and Neuro-Symbolic Vision-Language Reasoning Systems
Zhu Wang, Jesse Thomason, Robin Jia · 2022
For vision-and-language (VL) reasoning tasks, both fully connectionist, end-to-end methods and hybrid, neuro-symbolic methods have achieved high in-distribution performance.In which out-of-distribution settings does each paradigm excel?We investigate this question on both single-image and multi-image visual question-answering through four types of generalization tests: a novel segmentcombine test for multi-image queries, contrast set, compositional generalization, and crossbenchmark transfer.Vision-and-language endto-end (VLE2E) trained systems exhibit sizeable performance drops across all these tests.Neuro-symbolic (NS) methods suffer even more on cross-benchmark transfer from GQA to VQA, but they show smaller accuracy drops on the other generalization tests and their performance quickly improves by few-shot training.Overall, our results demonstrate the complementary benefits of these two paradigms, and emphasize the importance of using a diverse suite of generalization tests to fully characterize model robustness to distribution shift.Train A: There is at least 1 image with 2 bottles. Train B: Is the dark bottle on the table or not? Compositional GeneralizationContrast: There is less than 1 image with exactly 2 dark bottles on the table. Contrast