Quality-Based Pricing for Crowdsourced Workers
Jing Wang, Panagiotis G. Ipeirotis, Foster John Provost · SSRN Electronic Journal · 2013
The emergence of online paid crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), presents us huge opportunities to distribute tasks to human workers around the world, on-demand and at scale. In such settings, online workers can come and complete tasks posted by a company, and work for as long or as little as they wish. Given the absolute freedom of choice, crowdsourcing eliminates the overhead of the hiring (and dismissal) process. However, this exibility introduces a di erent set of ine ciencies: verifying the of every submitted piece of work is an expensive operation, which often requires the same level of e ort as performing the task itself. There are many research challenges that emerge in this paid-crowdsourcing setting. How can we ensure that the submitted work is accurate? How can we estimate the of the workers, and the of the submitted results? How should we pay online workers that have imperfect quality? We present a comprehensive scheme for managing of crowdsourcing processes: First, we present an algorithm for estimating the of the participating workers and, by extension, of the generated data. We show how we can separate systematic biases from unrecoverable errors and how to generate an unbiased worker quality measurement that can be used to objectively rank workers according to their performance. Next, we describe a pricing scheme that identi es the fair payment level for a worker, adjusting the payment level according to the contributed information by each worker. Our pricing policy, which pays workers based on their expected quality, reservation wage, and expected lifetime, estimates not only 1 the payment level but also accommodates measurement uncertainties and allows the workers to receive a fair wage, even in the presence of temporary incorrect estimations of quality. Our experimental results demonstrate that the proposed pricing strategy performs better than the commonly adopted uniform-pricing strategy. We conclude the paper by describing strategies that build on our control and pricing framework, to build crowdsourced tasks of increasingly higher complexity, while still maintaining a tight control of the process, even if we allow participants of unknown to join the process.