Weakly Supervised Automatic Transcription of Mouthings for Gloss-Based Sign Language Corpora
Oscar Koller, Hermann Ney, Richard Bowden · Surrey Research Insight Open Access (The University of Surrey) · 2014
In this work we propose a method to automatically annotate mouthings in sign language corpora, requiring no more than a simple gloss annotation and a source of weak supervision, such as automatic speech transcripts. For a long time, research on automatic recognition of sign language has focused on the manual components. However, a full understanding of sign language is not possible without exploring its remaining parameters. Mouthings provide important information to disambiguate homophones with respect to the manuals. Nevertheless most corpora for pattern recognition purposes are lacking any mouthing annotations. To our knowledge no previous work exists that automatically annotates mouthings in the context of sign language. Our method produces a frame error rate of 39 % for a single signer on the alignment task. Keywords:Sign Language, Mouthing, Lip Reading, Unsupervised Automatic Annotation 1.