A Fast Variational Approach for Learning Markov Random Field Language Models
Yacine Jernite, Alexander M. Rush · 2015
Language modelling is a fundamental building block of natural language processing. However, in practice the size of the vocabulary limits the distributions applicable for this task: specifi-cally, one has to either resort to local optimiza-tion methods, such as those used in neural lan-guage models, or work with heavily constrained distributions. In this work, we take a step to-wards overcoming these difficulties. We present a method for global-likelihood optimization of a Markov random field language model exploit-ing long-range contexts in time independent of the corpus size. We take a variational approach to optimizing the likelihood and exploit underly-ing symmetries to greatly simplify learning. We demonstrate the efficiency of this method both for language modelling and for part-of-speech tagging. 1.