Alternatives to Classic BM25-IDF based on a New Information Theoretical Framework
Weimao Ke · 2022 IEEE International Conference on Big Data (Big Data) · 2022
The IDF (Inverse Document Frequency) term weighting method is a classic treatment of a term’s significance in information retrieval and text analytics. IDF can be derived from the information-theoretic Kullback-Leibler (KL) Divergence and has given rise to competitive methods such as TF*IDF and Okapi BM25, which is the default scoring function of ElasticSearch. We developed a new information metric called DLITE and derived from it an alternative to IDF, namely iDL, for term weighting and scoring in ranked information retrieval. In a series of experiments we conducted on multiple benchmark Text REtrieval Conference (TREC) collections, iDL methods consistently outperformed BM25, a very competitive baseline, for ad hoc retrieval. We outline the theoretical properties of DLITE that support the effectiveness of iDL. As a general information measure, we expect DLITE to be applicable in many other areas of big-data analytics and machine learning where further research will be valuable.