SimpleQuestions Nearly Solved: A New Upperbound and Baseline Approach
Michael Petrochuk, Luke Zettlemoyer · 2018
The SimpleQuestions dataset is one of the most commonly used benchmarks for studying single-relation factoid questions.In this paper, we present new evidence that this benchmark can be nearly solved by standard methods.First, we show that ambiguity in the data bounds performance at 83.4%; many questions have more than one equally plausible interpretation.Second, we introduce a baseline that sets a new state-of-the-art performance level at 78.1% accuracy, despite using standard methods.Finally, we report an empirical analysis showing that the upperbound is loose; roughly a quarter of the remaining errors are also not resolvable from the linguistic signal.Together, these results suggest that the SimpleQuestions dataset is nearly solved.