A New Method for Identifying Low-Quality Data in Perceived Usability Crowdsourcing Tests: Differences in Questionnaire Scores
Yuhui Wang, Xuan Chen, Xuan Randy Zhou · International Journal of Human-Computer Interaction · 2023
To improve the accuracy of crowdsourced data, efficient elimination of low-quality data, commonly referred to as data cleaning, is a straightforward but powerful method. In this research, the effectiveness of various data cleaning methods was tested in two measuring environments. We used the score differences of three perceived usability questionnaires (SUS, UMUX, and mATM) to provide a new basis for accurately cleaning low-quality data. We accomplished this by observing the data cleaning effect of various score difference intervals. Our study ultimately showed that (1) significant differences in scores are found in different measurement settings, (2) completion time is a useful indicator for detecting low-quality data, (3) the correlation of the questionnaire after cleaning proves that a method combined with an inspection item pairs more strictly than a method using only completion time, and (4) a score difference of 30 points on the highly correlated perceived usability scale is a suitable cleaning threshold that is feasible for shorter questionnaires. Therefore, in the case of using a single questionnaire, a cleaning method combining the completion time and inspection item pairs can be used; we also recommend simultaneously using two standardized perceived usability questionnaires, a threshold score difference of 30 points between the two questionnaires can be used as a cleaning criterion.