CCQA: A New Web-Scale Question Answering Dataset for Model Pre-Training
Patrick Huber, Armen Aghajanyan, Barlas Oğuz, Dmytro Okhonko, Scott Yih, Sonal Gupta, Xilun Chen · Findings of the Association for Computational Linguistics: NAACL 2022 · 2022
We propose a novel open-domain questionanswering dataset based on the Common Crawl project.With a previously unseen number of around 130 million multilingual question-answer pairs (including about 60 million English data-points), we use our largescale, natural, diverse and high-quality corpus to in-domain pre-train popular language models for the task of question-answering.In our experiments, we find that our Common Crawl Question Answering dataset (CCQA) achieves promising results in zero-shot, low resource and fine-tuned settings across multiple tasks, models and benchmarks 1 .