SHACER: a Speech and Handwriting Recognizer
Edward Kaiser · 2005
the task domain of a multi-party, multimodal meeting focused on the creation of a whiteboard schedule chart, we have designed and implemented a general method of aligning handwriting and speech for capturing out-of-vocabulary terms, dynamically enrolling them in the system's recognition modules, and then using them to improve subsequent tracking and recognition. Our approach involves the use of an ensemble of syllable and phoneme recognizers for speech whose output is integrated with redundantly delivered handwriting recognition. We refer to our conceptual framework as Multimodal Out-Of- Vocabulary Recognition (MOOVR — pronounced mover). Within that framework this paper describes our Speech and HAndwriting reCognizER module (SHACER — pronounced shaker), which observes human-to-human spoken and handwritten interactions, analyzes them off-line and contributes improved recognitions to a record of the meeting in the form of a project schedule. We examine an example meeting and show how our technique corrects four of five label recognition errors including implicitly discovering the semantics of a handwritten abbreviation.