UnAnswGen: A Systematic Approach for Generating Unanswerable Questions in Machine Reading Comprehension
Hadiseh Moradisani, Fattane Zarrinkalam, Julien Serbanescu, Zeinab Noorian · 2024
This paper introduces a configurable software workflow to automatically generate and publicly share a dataset of multi-labeled unanswerable questions for Machine Reading Comprehension (MRC). Unlike existing datasets like SQuAD2.0, which do not account for the reasons behind question unanswerability, our method fills a critical gap by systematically transforming answerable questions into their unanswerable counterparts across various linguistic dimensions including entity swap, number swap, negation, antonym, mutual exclusion, and no information. These candidate unanswerable questions are evaluated using advanced MRC models to ensure their context-based unanswerability, with the final selection based on a majority consensus mechanism. Our approach addresses the scarcity of multi-labeled datasets like SQuAD2-CR, enabling comprehensive evaluation of MRC systems' ability to handle unanswerable queries and facilitating the exploration of solutions such as query reformulation. The resulting UnAnswGen dataset and associated software workflow are made publicly available to advance research in machine reading comprehension, offering researchers a standardized toolset for evaluating and enhancing MRC systems' robustness and performance.