You say "disaster", i say "no problem"
Rolf Molich, Jennifer McGinn, Nigel Bevan · 2013
This paper documents a consistent and considerable evaluator effect in the ratings of usability problem severity carried out by experienced usability professionals. The CUE-9 study, conducted in 2011, showed a reasonable level of rater agreement in problem identification, but the severity assigned to problems varied wildly from "not a problem" to "disaster". This paper documents the variations we observed and the need for a better rating scale and better training. The paper calls for more caution when using extreme ratings. Our results also show that simple attempts to fix the traditional rating scales may not work.