Improving consensus accuracy via Z-score and weighted voting
Hyun Joon Jung, Matthew Lease · 2011
Using supervised and unsupervised features individu-ally or together, we (a) detect and filter out noisy work-ers via Z-score, and (b) weight worker votes for consen-sus labeling. We evaluate on noisy labels from Amazon Mechanical Turk in which workers judge Web search relevance of query/document pairs. In comparison to a majority vote baseline, results show a 6 % error reduc-tion (48.83 % to 51.91%) for graded accuracy and 5% error reduction (64.88 % to 68.33%) for binary accuracy.