A Methodology for Evaluating Interaction Strategies of Task-Oriented Conversational Agents
Marco Guerini, Sara Falcone, Bernardo Magnini · 2018
In task-oriented conversational agents, more attention has been usually devoted to assessing task effectiveness, rather than to how the task is achieved.However, conversational agents are moving towards more complex and human-like interaction capabilities (e.g. the ability to use a formal/informal register, to show an empathetic behavior), for which standard evaluation methodologies may not suffice.In this paper, we provide a novel methodology to assess -in a completely controlled way -the impact on the quality of experience of agent's interaction strategies.The methodology is based on a within subject design, where two slightly different transcripts of the same interaction with a conversational agent are presented to the user.Through a series of pilot experiments we prove that this methodology allows fast and cheap experimentation/evaluation, focusing on aspects that are overlooked by current methods.