Subjective and Objective Evaluation of Conversational Agents in Learning Environments for Young Teenagers
Annika Silvervarg, Arne Jönsson · 2011
In this paper we present results from a study of subjective and objective evaluation metrics used to asses a conversational agent. Our study has been conducted in a school setting with students, aged 12 to 14 years old, who used a virtual learning environment that incorporates social conversation with a pedagogical agent. The subjective evaluation metrics capture the students ’ experiences of different aspects of the conversations, while the objective evaluation metrics are based on an analysis of the logs of the actual conversations. Our results show that there are no correlations between subjective and objective metrics that are supposed to measure the same aspects, for example, to what extent the system can correctly interpret and give responses to user utterances. They also indicate that different categories of users need to be considered, for example based on their attitude towards or engagement in the system.