Challenges and Important Aspects in Planning and Performing Evaluation Studies for Multimodal Dialogue Systems
Susanne Höllerer · 2002
In this paper I want to discuss the problems researchers face in trying to plan and carry out an evaluation study for multimodal systems -- particularly in qualifying the purpose of the testing, defining the intended user group for their application, arranging the testing setting and aligning the evaluation plan. It is my intention to show which aspects should be taken into account and which basic standards should be fulfilled. Furthermore, I provide two sections about points to consider in performing the study as well as in the analysis of the received data. Then I describe possible difficulties concerning the evaluation of (multimodal) systems and try to sketch longer term solutions. Finally, I list possible options on how to utilize the results of evaluations studies in further research.