Practical and efficient incorporation of syntactic features into statistical language models
Sanjeev P. Khudanpur, Ariya Rastrow · 2012
Automatic Speech Recognition (ASR) and Statistical Machine Translation (SMT), among other natural language processing applications, rely on a language model (LM) to provide a strong linguistic prior over word sequences of the often prohibitively large and complex hypothesis space of these systems. The language models deployed in most state-of-the-art ASR and SMT systems are n-gram models. Several statistical frameworks have been proposed to build more complex models and put (the syntactic structure of) language back into language modeling. Yet, n-gram models, despite being linguistically naive, are still favored, because estimating them from text is well understood, they are computationally efficient, and integrating them into ASR and SMT systems is straightforward. This dissertation proposes novel algorithms and techniques that make it practical to estimate and apply more complex language models in ASR and SMT tasks, in particular syntactic modes for speech recognition. While yielding significantly better performance than n-gram models, the syntactic structured language models (SLM) can not be efficiently trained on a large amount of text data due to the impractical size of the resulting model. A general information-theoretic pruning scheme is proposed to significantly reduce the size of the SLM while maintaining its prediction accuracy, therefore enabling efficient maximum likelihood estimation of the SLM parameters. The SLM, and other long-span language models, can not be directly applied during decoding or word lattice rescoring. Instead, these models are limited to an N-best rescoring framework, which as a search algorithm suffers from several known deficiencies and inefficiencies. Leveraging the theory and efficient algorithms for finite-state automata (FSA), an effective hill-climbing algorithm is developed for rescoring ASR lattices using long-span language models. It is shown that integrating the SLM into an ASR system in this manner significantly improves the WER over the computationally comparable N-best rescoring technique. Discriminative training of language models with long-span features, such as syntactic dependencies, triggers, and topic information, is limited to N-best lists for similar reasons. The FSA based hill climbing algorithm, proposed for the application of long-span models to ASR lattice rescoring, also paves the way for efficient discriminative training of long-span language models on speech lattices and thus, alleviates the shortcomings of N-best training. Long span language models—regardless of whether they are trained via maximum likelihood or trained discriminatively, and during both training and application—rely on auxiliary tools to extract nonlocal features. In the case of the SLM the syntactic features include a part of speech (POS) tagger and a parser for extracting features for each hypothesis during rescoring. These tools also slow down the application of such models in deployed systems. A general principle is presented wherein substructures common to multiple hypotheses are efficiently shared in the transition based structured prediction algorithms employed by these auxiliary tools. It is shown that the proposed substructure sharing algorithm results in substantial speedup when utilizing these tools in automatic speech recognition. The four methods and algorithms described above and detailed in this dissertation are evaluated using a state-of-the-art ASR system. It is demonstrated that these techniques could make the use of syntactically informed language models practical and hence, widespread. While this dissertation focuses on methods to efficiently apply syntactic language models to automatic speech recognition, many of the developed techniques may be used to efficiently utilize other complex language models in ASR and other applications such as machine translation.