Reliability of Perceived Usability Assessment via Crowdsourcing Platform: Retrospective Analysis and Novel Feedback Quality Inspection Method
Wang Yuhui, Lei Tian, Xinxiong Liu · International Journal of Human-Computer Interaction · 2020
This paper discusses the differences in perceived usability evaluations between crowdsourcing platform and laboratory testing. A retrospective evaluation measurement method was adopted. A SUS quality inspection method with two new sets of inspection items was established to more precisely observe certain low-quality feedback. We also tested a combination of the traditional check method (Items 2 and 3) and all inspection item pairs to observe the cleaning effect. The results of this study suggests that (i) crowdsourcing platform scores (using SUS) are higher than laboratory test scores, though their quality may not be as good, while the difference in CGS was only one grade. (ii) Using one inspection item pair is an acceptable cleaning method, but it may lead to over-cleaning. (iii) The internal relevance of the inspection item pairs is very high. The SUS quality inspection method proposed in this paper can improve the accuracy of perceived usability evaluations. (iv) The completion-time-based cleaning method may not require setting a maximum time in retrospective usability testing.