Cross-domain matching for automatic tag extraction across redundant handwriting and speech events
Edward Kaiser · 2007
In many types of natural human-human interactions people communicate important information redundantly across multiple communication modes, like saying what they handwrite during a presentation or discussion. To detect and benefit from such redundancies a computational understanding system must align the recognition outputs from different perceptual modes like handwriting and speech. Since the recognition domains of each mode differ, researchers refer to tasks like this as cross-domain matching. We describe how SHACER (our Speech and HAndwriting reCognizER) currently implements cross-domain matching, and compare that to an existing, formally optimal algorithm for this task. Successful alignment and recognition of such multimodal redundancies can be leveraged for automatic tagging of social interactions. These automatically generated tags can benefit retrieval techniques for non-textual documents recorded during computationally perceived social interactions.