Adaptation of multimodal input
Pedro Feiteira · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · 2011
This thesis is strongly coupled with the European project GUIDE (Gentle User Interfaces for Elderly Citizens) which intends to deliver a toolbox of adaptive multimodal interfaces to run on TV set-top boxes. The goal of this framework is to address some of the limitations and disabilities shown by elderly users and automatically adapt web-based applications to their needs also freeing the developers of the need of tackling accessibility issues. The User Trials Application is a multimodal application that was designed to perform user trials, which consisted on observing the users interacting with a multimodal system that supported multiple input/output modalities and capturing data about this interaction. This application allowed an high customization regarding tests including which interactive elements should appear on screen and their properties. A Wizard-of-Oz technique was used to empower the person running the tests and to allow a greater degree of control and information gathering. A second application developed, the User Initialization Application, constituted a prototype of the final version that is going to be present in the GUIDE framework, aimed for introducing the user to the system and input devices as well as gathering information about the user limitations so it could be assigned to a specific user model. The tests included in the prototype used various modalities such as speech and gestures. One of the main features of this application is the use of adaptation throughout the test sequence, changing properties such as volume, text size, color, among others. The third application discussed in this thesis is the GUIDE Fusion Core, responsible for user-adapted input combination. A frame-based algorithm was used to combine information and a weight-based approach to imprint adaptive behavior into it. Although the implementation of the GUIDE Fusion core is still in its early development, some focus was given to designing an evaluation framework capable of measuring, according to some metrics, the performance of the fusion core.