Beyond Heuristics: Multimodal Transformer for Chart Data Extraction
Yumeng Wang · 2024
Extracting data from charts is a crucial task for comprehending their content and facilitating subsequent tasks. Rule-based methods have conventionally been used to extract data from various chart types. However, these heuristic-driven solutions often require a considerable amount of human intervention and lack universal applicability. In our research, we present two innovative neural-based approaches aimed at transforming the chart data extraction field: 1) We have improved the current bottom-up bounding box detection approach for chart components by incorporating multiple transformer-based grouping modules, eliminating the requirement for heuristic rules. 2) Furthermore, we have drawn inspiration from the Question Answering paradigm to develop auto-generated tables. Rather than relying on time-consuming rules that focus on attributes such as position or color, our system identifies more primitive objects (comparable to answers) for each table cell (resembling questions). It is worth mentioning that our work provides impressive accuracy on established benchmarks and is adaptable to new chart designs. Additionally, our method is pioneering in its focus on autonomously learning chart extraction rules, which has the potential to pave the way for more universal chart data extraction in the future.