Target Suites for Evaluating the Coverage of Text Generators
John A. Bateman, Anthony F. Hartley · 2000
Our goal is to evaluate the grammatical coverage of the surface realization component of a natural language generation system by means of target suites.We consider the utility of re-using for this purpose test suites designed to assess the coverage of natural language analysis / understanding systems.We find that they are of some interest, in helping inter-system comparisons and in providing an essential link to annotated corpora.But they have limitations.First, they contain a high proportion of ill-formed items which are inappropriate as targets for generation.Second, they omit phenomena such as discourse markers which are key issues in text production.We illustrate a partial remedy for this situation in the form of a text generator that annotates its own output to an externally specified standard, the TSNLP scheme.