Sorting it All Out with Humans in the Loop.
Adam Marcus, Eugene W. Wu, David R. Karger, Samuel R. Madden, Rob Miller · Tiny Trans. Comput. Sci. · 2012
From an early age, we sort jellybeans with friends, placing watermelon flavor at the top of our list, and pushing licorice to the bottom. We contribute to larger-scale ranking operations when we rate a product in an online catalog. Given the prevalence of human-powered ranking in the digital realm, it is important to study the most effective user interfaces and algorithms for eliciting rankings from humans. In experiments [2] involving Mechanical Turk [1] workers, we found that average ratings (e.g., “On a scale from one to seven”) have a cost linear in the input and result in high-accuracy rankings. Comparison-based sorts (e.g., “Which of these pairs of glasses is cooler?”) have a cost up to quadratic in the input, but result in perfect rankings. We found that a hybrid of the two, where one rates the input and then runs comparisons on the nearly ranked data as a budget allows, is both costand accuracy-effective. Lest we leave you with the foul-tasting notion that people are simply binary comparator-computing cogs in an algorithm-powered machine, there’s more to humans than that! Humans are effective at batch-processing data in ways that machines are not. We take advantage of this ability to reduce the complexity of both ratingand comparison-based rankings by showing crowd workers five to ten items at a time and having them rate or compare all of the items. There is more to this story, but it’s time for a snack! BODY When ranking with humans, ratings are cheap and nice. To improve quality, please compare, for a price.