An Analysis of Dataset Overlap on Winograd-Style Tasks
Ali Reza Emami, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung · 2020
The Winograd Schema Challenge (WSC) and variants inspired by it have become important benchmarks for common-sense reasoning (CSR).Model performance on the WSC has quickly progressed from chance-level to near-human using neural language models trained on massive corpora.In this paper, we analyze the effects of varying degrees of overlap between these training corpora and the test instances in WSC-style tasks.We find that a large number of test instances overlap considerably with the corpora on which state-of-the-art models are (pre)trained, and that a significant drop in classification accuracy occurs when we evaluate models on instances with minimal overlap.Based on these results, we develop the KNOWREF-60K dataset, which consists of over 60k pronoun disambiguation problems scraped from web data.KNOWREF-60K is the largest corpus to date for WSC-style common-sense reasoning and exhibits a significantly lower proportion of overlaps with current pretraining corpora.