Discriminative, Syntactic Language Modeling through Latent SVMs
Colin Cherry, Chris Quirk · 2008
We construct a discriminative, syntactic lan-guage model (LM) by using a latent support vector machine (SVM) to train an unlexical-ized parser to judge sentences. That is, the parser is optimized so that correct sentences receive high-scoring trees, while incorrect sentences do not. Because of this alternative objective, the parser can be trained with only a part-of-speech dictionary and binary-labeled sentences. We follow the paradigm of dis-criminative language modeling with pseudo-negative examples (Okanohara and Tsujii, 2007), and demonstrate significant improve-ments in distinguishing real sentences from pseudo-negatives. We also investigate the re-lated task of separating machine-translation (MT) outputs from reference translations, again showing large improvements. Finally, we test our LM in MT reranking, and investi-gate the language-modeling parser in the con-text of unsupervised parsing. 1