Comparing test-suite based evaluation and corpus-based evaluation of a wide-coverage grammar for English
Rashmi Prasad, Anoop Sarkar · 2001
In this paper we present our experiences in the evaluation of a wide-coverage grammar of English: the XTAG English grammar. We give a brief history of previous evaluations done using the XTAG grammar and then describe a pair of new evaluations done on a corpus of weather reports and the CSLI LKB test suite. Based on these experiments, we discuss the differing merits of naturally occurring corpora and test suites in the evaluation of wide-coverage grammars. 1. Introduction The term wide-coverage when applied to natural language parsing has two meanings: one is that of grammatical coverage: how many linguistic phenomena does the grammar handle; the second is that of parsing coverage: for how many sentences from a naturally occurring corpora of text does a parser produce an appropriate derivation. Evaluation methods such as test-suite based evaluation handle the former notion of coverage, while corpus-based metrics (Harrison et al., 1991) attempt to handle the latter notion of coverage....