Stop Word Lists in Free Open-source Software Packages
Joel Nothman, Hanmin Qin, Roman Yurchak · 2018
Open-source software (OSS) packages for natural language processing often include stop word lists.Users may apply them without awareness of their surprising omissions (e.g.hasn't but not hadn't) and inclusions (e.g.computer), or their incompatibility with particular tokenizers.Motivated by issues raised about the Scikitlearn stop list, we investigate variation among and consistency within 52 popular English-language stop lists, and propose strategies for mitigating these issues.