Language Model Estimation from a Stochastically Tagged Corpus
Shinsuke Mori, Tetsuro Sasada, Graham Neubig · Journal of Natural Language Processing · 2011
In this paper, first we propose a language model based on pairs of word and input sequence.Then we propose the notion of a stochastically tagged corpus to cope with tag estimation errors.The experimental results we conducted using kana-kanji converters showed that our ideas, the language model based on pairs of word and input sequence and the notion of a stochastically tagged corpus, both improved the accuracy.Therefore we conclude that the language model based on pairs and the notion of a stochastically tagged corpus are effective in language modeling for the kana-kanji conversion task.