Effective active learning for complex natural language processing tasks
Florian Laws · OPUS Publication Server of the University of Stuttgart (University of Stuttgart) · 2013
Supervised machine learning is a widely used approach to natural language processing tasks. However, supervised learning needs large amounts of labeled training data, which needs to be annotated in a time-consuming and expensive process. Active learning is a strategy to reduce this annotation effort by setting up an interactive process in which the machine learning system iteratively selects data for annotation. By selecting only data that the system considers informative, this strategy promises a significant reduction of data that is needed for training. In this thesis, we investigate the application of active learning to key natural language processing tasks. We investigate selection strategies for “informative” training examples for two key NLP tasks: named entity recognition and coreference resolution. We show that active learning can deliver a large reduction in annotation effort for these NLP tasks. However, in cases of unfortunate initialization, active learning can suffer from slow learning progress on infrequent classes: the missed cluster effect. We show that active learning can be made resilient against this phenomenon by co-selecting examples that occur together in a natural context (e.g. a sentence). We also apply this strategy to selection of examples for coreference annotation and could demonstrate for the first time a successful active learning approach to coreference resolution. We also monitor training progress during data annotation. We investigate a method to estimate performance without additional labeled test data. While this method is not reliable for stopping at a performance threshold, we can use it to define effective criteria to stop when performance for a given system and given dataset is close to optimal. Finally, we investigate crowdsourcing as a complementary cost reduction approach that aims to reduce the per-example cost by outsourcing annotation over the web. We propose strategies to mitigate the higher mistake rates of crowdsourcing annotators and present a successful combination of active learning with crowdsourcing.