Stopword Removal
Kripabandhu Ghosh, Arnab Bhattacharya · 2017
Stopword removal has traditionally been an integral step in information retrieval pre-processing. In this paper, we question the utility of this step in retrieving relevant documents for verbose queries on standard datasets. We show that stopword removal does not lead to noticeable difference in retrieval performance as opposed to not removing them. We observe this phenomenon in 7 FIRE test collections for 4 Indian languages, Bangla, Hindi, Gujarati and Marathi, as well as for European languages such as Czech (CLEF 2007) and Hungarian (CLEF 2005 to 2007). Since these languages are inflective, the stopword lists are not significant. More interestingly, for languages such as English (TREC678 Ad Hoc) and French (CLEF 2005 to 2007), stopword removal leads to a statistically significant drop in performance. This is due to using a generic stopword list that does not suit in many document retrieval tasks.