An architecture and interaction techniques for handling ambiguity in recognition-based input
Gregory D. Abowd, Jennifer C. Mankoff · 2001
It is difficult to build applications that effectively use recognizers, in part because of lack of toolkit-level support for dealing with recognition errors. This dissertation presents an architecture that addresses that problem. Recognition technologies such as speech, gesture, and handwriting recognition, have made great strides in recent years. By providing support for more natural forms of communication, recognition can make computers more accessible. Such “natural” interfaces are particularly useful in settings where a keyboard and mouse are not available, as in very large or very small displays, and in mobile and ubiquitous computing. However, recognizers are error-prone: they may not interpret input as the user intended. This can confuse the user, cause performance problems, and result in brittle interaction dialogues. The major contributions of this thesis are: (1) A model of recognition that uses ambiguity to keep track of errors and mediation to correct them. This model can be applied to any recognition-based interface found in the literature, all of which include some type of support for mediation of recognition errors. (2) A corresponding toolkit architecture that uses this model to represent recognition results. The toolkit architecture benefits programmers who build recognition-based applications, by providing a separation of recognition and mediation from application development. At a high level, mediation intervenes between the recognizer and the application in order to resolve ambiguity. This separation of concerns allows us to create re-usable solutions to mediation similar to the menus, buttons, and interactors provided in any GUI toolkit. The separation of recognition also leads to the ability to adapt mediation to situations which do not seem to be recognition-based, but where some problem in interaction causes the system to do something other than what the user intended (our ultimate definition of error). Finally, the separation of mediation allows us to develop complex mediation interactions independent of both the source of ambiguity and the application.