Research on Language Modeling Based on Dirichlet Distribution for Information Retrieval Technology
LI Zhou-jun · Computer Engineering and Science · 2008
The word burstiness phenomenon is not taken into account in the unigram language model based on multinomial distribution,but the language model based on Dirichlet distribution can do it well.This paper analyses and discusses several language models based on Dirichlet distribution,moreover,the DCM model is used as the basic model for document modeling and query modeling,the similarity between document and query is measured by KL-divergence.Experimental results in the TREC data sets show that the DCM language model can improve the mean precision compared with the basic model.