Stemming and its effects on TFIDF ranking.
Mark Kantrowitz, Behrang Mohit, Vibhu O. Mittal · 2000
High precision IR is often hard for a variety of reasons; one of these is the large number of morphological variants for any given term. To address some of the issues arising from a mismatch between different word forms used in the queries and the relevant documents, researchers have long proposed the use of various stemming algorithms to reduce terms to a common base form. Stemming, in general, has not been an unmitigated success in improving IR. This poster argues that stemming can help in certain contexts and that an empirical investigation of the relationship between stemming performance and retrieval performance can be valuable. We extend previously reported work on stemming and IR (e.g., [2, 4, 5]) by using a novel, dictionary based “perfect” stemmer, which can be parametrized for different accuracy and coverage levels. This allows us to measure changes in IR performance for any given change in stemming performance on a given data set. To place this work in context, we discuss an empirical evaluation of stemming accuracy for three stemming algorithms – including the widely used Porter algorithm [9]. Section 2 briefly discusses the three variants of stemming, and presents experimental evidence for their relative coverage and accuracy. Section 3 discusses the use of these stemmers for IR and presents some of our findings. Finally, Section 4 concludes with a discussion of our results and possible future directions.