Lexical normalization model for noisy SMS text
Greety Jose, Nisha S. Raj · 2014
In day to day life, digital mediated interactions and communications being an important constituent. The expeditious growth of electronic communications such as E-mails, micro blogs, SMS and chats etc has fabricated extensively noisy forms of text. It predominantly in young urbanités. The tremendous growth of noises in text are due to a variety of factors, such as the small number of characters allowed per text messages (160 characters is allowed per SMS and 140 characters allowed per tweets), inventing new abbreviations, using non standard orthographic forms, phonetic substitution etc. In this paper we introduce a lexical normalization model for cleaning the noisy texts. The normalization is based on the channelized database. The model will capture the user interaction for improving the model accuracy. Precursory evaluation shows that the channel model will normalize the noisy word to their standard peer with 97.5 % accuracy.