Assessing the Utility of the System Usability Scale for Evaluating Voice-based User Interfaces
Debjyoti Ghosh, Pin Sym Foong, Shan Zhang, Shengdong Zhao · 2018
Voice-based User Interfaces (VUIs) challenge our existing conceptions of usability since the standardized evaluation tools we use were typically developed for interfaces with visual feedback, whereas VUI's have predominantly eyes-free interactions. We experimented with the use of a well-validated tool, the System Usability Scale (SUS), to evaluate two existing, commercially available VUIs. We administered the SUS to 12 participants after they completed a set of scenario tasks on Amazon's Alexa and Apple's Siri. The results were consistent with previous studies comparing subjective rating scores and SUS, suggesting that the SUS is a valid evaluation tool for VUIs. Additionally, despite large, significant differences in adjective scale ratings and SUS scores, both systems performed similarly on the learnability items. We conclude with recommendations on the use of the SUS for evaluating VUIs.