Methodology for Reliable Schema Development and Evaluation of Manual Annotations
Petra Saskia Bayerl, Ulrike B. Gut, Karsten Ingmar Paul · 2003
The quality of manual annotations of linguistic data depends on the use of reliable coding schemas as well as on the ability of human annotators to handle them appropriately. As is well known from a wide range of previous experiences annotations using highly complex coding schemas often lead to unacceptable annotation quality. Reducing complexity might make schemas easier to handle, but in this way valuable information needed for more sophisticated applications is excluded as well. In order to deal with this problem, we developed a systematic approach to schema development, which allows for developing coding schemas for fine-grained semantic annotations while systematically securing the quality of such annotations. For illustration, we present examples from two projects where text and speech data are annotated.