Empirical Analysis of Aggregation Methods for Collective Annotation
Ciyang Qing, Ulle Endriss, Raquel Fernández, Justin Kruger · UvA-DARE (University of Amsterdam) · 2014
We investigate methods for aggregating the judgements of multiple individuals in a linguistic annotation task into a collective judgement.We define several aggregators that take the reliability of annotators into account and thus go beyond the commonly used majority vote, and we empirically analyse their performance on new datasets of crowdsourced data.