Modeling of term-distance and term-occurrence information for improving n-gram language model performance
Tze Yuang Chong, Rafael E. Banchs, Eng Siong Chng, Haizhou Li · Meeting of the Association for Computational Linguistics · 2013
In this paper, we explore the use of distance and co-occurrence information of word-pairs for language modeling. We attempt to extract this information from history-contexts of up to ten words in size, and found it complements well the n-gram model, which inherently suffers from data scarcity in learning long history-contexts. Evaluated on the WSJ corpus, bigram and trigram model perplexity were reduced up to 23.5% and 14.0%, respectively. Compared to the distant bigram, we show that word-pairs can be more effectively modeled in terms of both distance and occurrence.