Enhancing Question Answering on Charts Through Effective Pre-training Tasks
Ashim Gupta, Vivek Gupta, Shuo Zhang, Yujie He, Ning Zhang, Shalin Shah · 2024
To completely understand a document, the use of textual information is not enough.Understanding visual cues, such as layouts and charts, is also required.While the current state-ofthe-art approaches for document understanding (both OCR-based and OCR-free) work well, we have not found any other works conducting a thorough analysis of their capabilities and limitations.Therefore, in this work, we address the limitation of current VisualQA models when applied to charts and plots.To investigate shortcomings of the state-of-the-art models, we conduct a comprehensive behavioral analysis, using ChartQA as a case study.Our findings indicate that existing models underperform in answering questions related to the chart's structural and visual context, and also numerical information.To address these issues, we propose three simple pre-training tasks that enforce the existing model in terms of structural-visual knowledge, and its understanding of numerical questions.We evaluate our pre-trained model (called MatCha-v2) on three chart datasets -both extractive and abstractive question datasets -and observe that it achieves an average improvement of 1.7% over the baseline model.