Beyond independent agreement: A tournament selection approach for quality assurance of human computation tasks
Yuan Sun, Shourya Roy, Greg Little · 2011
Quality assurance remains a key topic in human computation research field. Prior work indicates independent agreement is effective for low difficulty tasks, but has limitations. This paper addresses this problem by proposing a tournament selection based quality control process. The experimental results from this paper show that the human are better at identifying the correct answers than producing them themselves.