Evaluating the Impact of Coder Errors on Active Learning
Ines Rehbein, Josef Ruppenhofer · Publication Server of the Institute for German Language (Institute for German Language) · 2011
Active Learning (AL) has been proposed as a technique to reduce the amount of annotated data needed in the context of supervised classification.While various simulation studies for a number of NLP tasks have shown that AL works well on goldstandard data, there is some doubt whether the approach can be successful when applied to noisy, real-world data sets.This paper presents a thorough evaluation of the impact of annotation noise on AL and shows that systematic noise resulting from biased coder decisions can seriously harm the AL process.We present a method to filter out inconsistent annotations during AL and show that this makes AL far more robust when applied to noisy data.