DICIT: Evaluation of a Distant-talking Speech Interface for Television
Timo Sowa, Fiorenza Arisio, Luca Cristoforetti · 2010
The EC-funded project DICIT developed distant-talking interfaces for interactive TV.The final DICIT prototype system processes multimodal user input by speech and remote control.It was designed to understand both natural language and command-and-control-style speech input.We conducted an evaluation campaign to examine the usability and performance of the prototype.The task-oriented evaluation involved naïve test persons and consisted of a subjective part with a usability questionnaire and an objective part.We used three groups of objective metrics to assess the system: one group related to speech component performance, one related to interface design and user awareness, and a final group related to task-based effectiveness and usability.These metrics were acquired with a dedicated transcription and annotation tool.The evaluation revealed a quite positive subjective assessments of the system and reasonable objective results.We report how the objective metrics helped us to determine problems in specific areas and to distinguish design-related issues from technical problems.The metrics computed over modality-specific groups also show that speech input gives a usability advantage over remote control for certain types of tasks.