SciDQA: A Deep Reading Comprehension Dataset over Scientific Papers
Shruti Singh, Nandan Sarkar, Arman Cohan · 2024
Scientific literature is typically dense, requiring significant background knowledge and deep comprehension for effective engagement.We introduce SCIDQA, a new dataset for reading comprehension that challenges LLMs for a deep understanding of scientific articles, consisting of 2,937 QA pairs.Unlike other scientific QA datasets, SCIDQA sources questions from peer reviews by domain experts and answers by paper authors, ensuring a thorough examination of the literature.We enhance the dataset's quality through a process that carefully filters out lower quality questions, decontextualizes the content, tracks the source document across different versions, and incorporates a bibliography for multi-document question-answering.Questions in SCIDQA necessitate reasoning across figures, tables, equations, appendices, and supplementary materials, and require multi-document reasoning.We evaluate several open-source and proprietary LLMs across various configurations to explore their capabilities in generating relevant and factual responses.Our comprehensive evaluation, based on metrics for surface-level similarity and LLM judgements, highlights notable performance discrepancies.SCIDQA represents a rigorously curated, naturally derived scientific QA dataset, designed to facilitate research on complex scientific text understanding.