TYCOON: Theoretical Framework and Software Tools for Multimodal Interfaces
Jean‐Claude Martin · 1997
We define a modality as a process analyzing and producing chunks of information. For instance, a speech recognition modality analyses speech signals and produces the labels of recognized words. Several multimodal interfaces combining such modalities have already been developed (IMMI’95, CMC’95). To take benefit out of them so as to advance research and implementation of multimodal interfaces, coherent theoretical and software tools are needed. From the “theoretical ” point of view, the development of multimodal interfaces addresses several issues (Maybury 91, Dowell 95): content selection (“what to say”), modality allocation (“which modality to say it”), modality realization (“how to say it in that modality”) and modality combination. This paper deals with the “modality combination ” issue. A multimodal interface developer has to know how to combine modalities and why this combination may improve the interaction. Yet existing frameworks for human-computer interfaces do not answer these two questions. Instead, they deal with the relation between the modes