Analyzing the Performance of Message Understanding Systems
Amit Bagga, Alan W. Biermann · 1998
In this paper we describe a method of classifying facts (information) into categories or levels; where each level signifies a different degree of difficulty of extracting the fact from a piece of text containing it. We use this method to analyze the performances of three MUC systems (BBN, NYU, and SRI) based on their ability to extract a set of standard facts (at different levels) from two different MUC domains. This analysis is then extended to analyze the role of coreferencing in the performance of message understanding systems. Introduction Earlier, in (Bagga 1997), we had described a method of classifying facts (information) into categories or levels; where each level signifies a different degree of difficulty of extracting the fact from a piece of text containing it. We then used this method to evaluate three different Message Understanding Conference (MUC) domains for information extraction tasks by assigning, to each domain, a "domain number" based on the levels of a set of "st...