Normalizing microtext
Zhenzhen Xue, Dawei Yin, Brian D. Davison · 2011
The use of computer mediated communication has resulted in a new form of written text—Microtext—which is very differ-ent from well-written text. Tweets and SMS messages, which have limited length and may contain misspellings, slang, or abbreviations, are two typical examples of microtext. Micro-text poses new challenges to standard natural language pro-cessing tools which are usually designed for well-written text. The objective of this work is to normalize microtext, in order to produce text that could be suitable for further treatment. We propose a normalization approach based on the source channel model, which incorporates four factors, namely an orthographic factor, a phonetic factor, a contextual factor and acronym expansion. Experiments show that our approach can normalize Twitter messages reasonably well, and it outper-forms existing algorithms on a public SMS data set.