Evaluation of Coreferences and Coreference Resolution Systems
Amit Bagga · 1998
Reference Resolution (or coreference) is an important tool for information extraction systems (systems which extract facts regarding "who did what to whom, when, and where"). Because of its importance to information extraction, coreference was added as a formal task for the Sixth Message Understanding Conference (MUC-6) [MUC-6] held in 1995. MUC-6 was successful in directing the attention of natural language processing research to the coreference task and in providing a framework within which one could evaluate a coreference system. However, the precision and recall statistics used by MUC-6 give little insight, both, into the actual performance of the coreference systems, and the difficulty of the coreference task. In this paper we present a structured framework for evaluating coreferences and coreference resolution systems. Based upon the coreference task definition of MUC-6 we break the coreference task into 11 classes (i.e. any coreference falls into one of these 1...