Human communication with information systems
James L. Flanagan · The Journal of the Acoustical Society of America · 2000
As networked information systems increase in complexity, natural modes of communication assume a greater role in the design of human–machine interfaces. Modalities of sound, sight, and touch are largely favored by humans, with conversational interaction being a central medium for information exchange. The distinguished research career of Professor Kenneth N. Stevens, celebrated in this special session on speech communication, has contributed directly to the basic understanding that supports current capabilities in voice interaction. While as yet primitive in comparison to human versatility, the technologies for sound, in concert with those for sight and touch, can be prudently applied to transcend the functionalities of traditional mouse and keyboard. This report describes a research effort in simultaneous utilization and fusion of sound, sight, and touch signaling in a networked collaborative environment. The conversational interaction depends upon automatic speech recognition, speech synthesis, and ‘‘hands-free’’ sound capture. Automatic face recognition, visual gesture, and eyetracking serve the sight modality. A force-feedback tactile glove provides manual gesture and grasp of virtual objects. And, intelligent software agents fuse and manage the multiple modes to develop a context-aware estimate of user intent. An experimental system utilizing these interface modes is applied to crisis management, and is demonstrated by videotape. [Components of this research were supported by National Science Foundation Contract Nos. IRI-96-18854 (STIMULATE) and IIS-98-72995 (KDI), DARPA Contract No. N66001-96-C-8510 (DISCIPLE), the Intel Corporation, Texas Instruments, and the Rutgers Center for Advanced Information Processing.]