Gaussian Process Classification and Active Learning with Multiple Annotators
Filipe Rodrigues, Francisco C. Pereira, Bernardete Ribeiro · 2014
Learning from multiple annotators took a valu-able step towards modeling data that does not fit the usual single annotator setting, since multi-ple annotators sometimes offer varying degrees of expertise. When disagreements occur, the es-tablishment of the correct label through trivial so-lutions such as majority voting may not be ad-equate, since without considering heterogeneity in the annotators, we risk generating a flawed model. In this paper, we generalize GP classi-fication in order to account for multiple annota-tors with different levels expertise. By explicitly handling uncertainty, Gaussian processes (GPs) provide a natural framework for building proper multiple-annotator models. We empirically show that our model significantly outperforms other commonly used approaches, such as majority voting, without a significant increase in the com-putational cost of approximate Bayesian infer-ence. Furthermore, an active learning method-ology is proposed, which is able to reduce anno-tation cost even further. 1.