Overview of evaluation in speech and natural language processing
Lynette Hirschman, Henry S. Thompson · 1997
Introduction to Evaluation Terminology and Use We can broadly distinguish three kinds of evaluation, appropriate to three different goals. 1. Adequacy Evaluation This is determination of the fitness of a system for a purpose---will it do what is required, how well, at what cost, etc. Typically for a prospective user, it may be comparative or not, and may require considerable work to identify a user's needs. One model is consumer organizations which publish the results of tests on, e.g., cars or appliances, and identify best buys for certain price-performance targets. This also goes by the names evaluation and evaluation proper. 476 Chapter 13: Evaluation 2. Diagnostic Evaluation This is production of a system performance profile with respect to some taxonimization of the space of possible inputs. It is typically used by system developers, but sometimes offered to end-us