Getting What You Paid For: Assessing Participant Experience Parameters for Amazon Mechanical Turk (MTurk) Workers on Survey Response Quality
M. Kumar, Doyeon Kim, Paul Weisgarber, Joseph S. Valacich, James L. Jenkins · 2023
MTurk is a powerful and widely used method for completing online Human Intelligence Tasks (HITs) such as website testing or completing psychological surveys. Researchers typically use various platform-provided work experience parameters to choose qualified and unqualified participants. Conventional wisdom suggests that participants with greater experience and historically high acceptance rates (i.e., higher-rated) will generate better quality data than their peers with lower experience and low acceptance rates (i.e., lower-rated). We examine the limits of this assumption by comparing responses and engagement behaviors between higher-rated, experienced (HE) participants and lower-rated, inexperienced (LI) participants while answering online surveys. We administered an online survey where participants first answered questions related to their MTurk account profile and then answered questions related to their personality. LI participants provide more inaccurate responses when answering factual questions (i.e., higher error rates) compared to HE participants. They also exhibit lower engagement behaviors when answering personality-related questions, resulting in marginally lower reliability scores for survey constructs. We are encouraged by the systematic differences we observed between the two populations and urge researchers to consider our findings before selecting optimal work experience parameters.