Evaluating Robustness and Diversity in Visual Question Answering Using Multimodal Large Language Models
Xixi Ga, Wenjie Liu, Tongyu Zhu, Shan Kou, Meishen Liu, Hu Yue · 2024
The increasing complexity of tasks requiring both visual and textual understanding has driven the development of advanced models capable of handling multimodal data. A novel evaluation of robustness and diversity in Visual Question Answering (VQA) was introduced through the application of multimodal models, specifically LLaMA, across a range of diverse datasets and challenging conditions. LLaMA demonstrated strong performance not only in standard benchmarks but also in handling adversarial attacks, out-of-distribution inputs, and noisy environments, showcasing its adaptability in unpredictable scenarios. The study highlighted the role of modular visual encoders and cross-modal attention mechanisms in maintaining model coherence and accuracy under varying degrees of input perturbation. Through rigorous comparative testing, the research underscored the importance of sophisticated model architectures for improving generalization capacity and robustness in VQA tasks. Key findings emphasized the strengths of LLaMA in maintaining performance under challenging conditions while also identifying areas for potential improvements in generalization across unfamiliar domains.