The Relationship between Answer Ranking and User Satisfaction in a Question Answering System
Tomoharu Kokubu, Tetsuya Sakai, Yoshimi Saito, Hideki Tsutsui, Toshihiko Manabe, Makoto Koyama, Hiroko Fujii · NTCIR · 2005
Although research in effective Question Answering (QA) has become active in recent years, it was not clear how system effectiveness affects user satisfaction in a practical QA environment. We therefore considered two practical environments in which QA may be useful (namely, Desktop and Mobile) and conducted a questionnaire survey for each environment. The objective was to clarify the relationship between the rank of a correct answer and the Proportion of Satisfied Users (PSU). Results show that, while the PSU curve resembles that of Reciprocal Rank for the Desktop case, it is almost proportional to the rank for the Mobile case. That is, whether Reciprocal Rank accurately models user satisfaction seems to depend on how the ranked answers are presented to the user. Based on our findings, we claim that QA system developers should set a goal in terms of the distribution of correct answers over ranks, instead of a single Mean Reciprocal Rank value, in order to satisfy the users.