Clustering word category based on binomial posteriori co-occurrence distribution
Masafumi Tamoto, Takeshi Kawabata · 2002
This paper describes a word clustering technique for stochastic language modeling and reports experimental evidence for its validity. The binomial posteriori distribution (BPD) distance measurement between words is introduced. It is based on word co-occurrency and reliability. We plan to consider a practical application of this clustering technology by utilizing each cluster as a Markov state in the construction of a word prediction model.