End-to-End Evaluation of Multimodal Dialogue Systems – can we Transfer Established Methods?
Nicole Beringer, Katerina Louka, Victoria Penide-Lopez, Uli Türk · 2002
The goal of this paper is to define a methodology for the end-to-end evaluation of the multimodal dialogue system SmartKom along the lines of the DARPA guidelines for spoken dialogue systems.The methodology consists of an extended framework for the evaluation of a multimodal dialogue system, evaluation metrics for its various components, and an approach to compare the user satisfaction with the system's technical performance.