Utilizing passage-based language models for document retrieval
Michael Bendersky, Oren Kurland · 2008
Abstract. We show that several previously proposed passage-based doc-ument ranking principles, along with some new ones, can be derived from the same probabilistic model. We use language models to instantiate spe-ci c algorithms, and propose a passage language model that integrates information from the ambient document to an extent controlled by the estimated document homogeneity. Several document-homogeneity mea-sures that we propose yield passage language models that are more ef-fective than the standard passage model for basic document retrieval and for constructing and utilizing passage-based relevance models; the latter outperform a document-based relevance model. We also show that the homogeneity measures are eective means for integrating document-query and passage-query similarity information for document retrieval.