Large Language Models Based Stemming for Information Retrieval: Promises, Pitfalls and Failures
Shuai Wang, Shengyao Zhuang, Guido Zuccon · 2024
Text stemming is a natural language processing technique that is used to reduce words to their base form, also known as the root form. In Information Retrieval (IR), stemming is used in keyword-based matching pipelines to normalise text before indexing and query processing to improve subsequent matching between document and query keywords. The use of stemming has been shown to often improve the effectiveness of keyword-matching models such as BM25. However, traditional stemming methods, focusing solely on individual terms, overlook the richness of contextual information.