Studying the effects of noisy text on text mining applications
Lipika Dey, Shamsul Haque · 2009
Text mining aims at deriving high quality information from text in an automated way. Text mining applications rely on Natural Language Processing (NLP) tools like tagger, parser etc. to locate and retrieve relevant information in an application specific manner. Most of these NLP tools however have been designed to work on clean and grammatically correct text. Presently, many organizations are interested to derive information from informally written text that is generated as a result of human communication through emails, or blog posts, web-based reviews etc. These texts are highly noisy and often found to contain mixture of languages. In this study we present some analysis on how noise introduced due to incorrect English affects the performance of some of the NLP tools and thereafter the text mining applications. The text mining application that we focus on is opinion mining. Opinion mining is the most significant text mining application that has to deal with noisy text generated in an unregulated fashion by users.