Linguistically Motivated Bigrams in Part-of-Speech Tagging of Language Corpora
Karel Oliva, Pavel Květoň · 2002
After some discussion concerning the issues of corpus representativity in the first paragraphs, this paper presents a simple yet in practice very efficient technique serving for automatic detection of those positions in a Part-of-Speech tagged corpus where an error is to be suspected. The approach is based on the idea of creating and then applying a set of "invalid bigrams", i.e. of pairs of adjacent Part-of-Speech tags which constitute an incorrect configuration in a tagged text of a particular language (in English, e.g., the bigram [ARTICLE, FINITE VERB]). Further, the paper describes the generalization of the "invalid bigrams " into a certain set of "invalid n-grams", for any natural n, which indeed provides a powerful tool for error detection in a corpus. Some implementation issues are also presented, as well as evaluation of results of the approach when used for error detection in the NEGRA corpus of German. Finally, general implications for the quality of results of statistical taggers are discussed. Illustrative examples in the text are taken mainly from German, and hence at least a basic command of this language would be helpful for their understanding – due to the complexity of the necessary accompanying explanation, the examples are neither glossed nor translated. However, the central ideas of the paper should be understandable also without any knowledge of German. 1. Errors in Part-of-Speech Tagged Corpora The importance of correctness (absence of errors) of language resources in general and of tagged