Explaining How to Interpret a Usability Survey May Enhance Internal Reliability Without Negatively Influencing Responses
Ian W. T. Robertson, Xiaoxuan Cheng, Philip T. Kortum · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · 2025
A great challenge to employing questionnaires is ensuring that respondents understand them. One possible strategy is to explain the meaning of items but there is a risk in biasing respondents’ answers. In this study, we assessed how explaining the meaning of items to respondents might affect their responses and the validity/reliability of the Positive System Usability Scale (PSUS). We employed straightforward, general explanations that we felt matched the intended meaning of the items. We found a statistically significant difference between the Explanations and No Explanations groups, but we judged this effect as not practically significant. We found the validity and reliability of the questionnaires from both groups to be acceptable. Finally, the reliability of the PSUS questionnaires given to the Explanations group was systematically higher than the No Explanations group, suggesting that correctly explaining items may enhance the reliability of questionnaires, perhaps by enhancing respondents’ understanding of the questionnaire items.