PAGED: A Benchmark for Procedural Graphs Extraction from Documents
Weihong Du, Wenrui Liao, Hongru Liang, Wenqiang Lei · 2024
Automatic extraction of procedural graphs from documents creates a low-cost way for users to easily understand a complex procedure by skimming visual graphs.Despite the progress in recent studies, it remains unanswered: whether the existing studies have well solved this task (Q1) and whether the emerging large language models (LLMs) can bring new opportunities to this task (Q2).To this end, we propose a new benchmark PAGED, equipped with a large high-quality dataset and standard evaluations.It investigates five state-of-the-art baselines, revealing that they fail to extract optimal procedural graphs well because of their heavy reliance on hand-written rules and limited available data.We further involve three advanced LLMs in PAGED and enhance them with a novel self-refine strategy.The results point out the advantages of LLMs in identifying textual elements and their gaps in building logical structures.We hope PAGED can serve as a major landmark for automatic procedural graph extraction and the investigations in PAGED can provide valuable insights into the research on logical reasoning among non-sequential elements.The code and dataset are available in https://github.com/SCUNLP/PAGED.