A Broad-Coverage Normalization System for Social Media Language
Fei Liu, Fuliang Weng, Jiang Xiao · 2012
Social media language contains huge amount and wide variety of nonstandard tokens, cre-ated both intentionally and unintentionally by the users. It is of crucial importance to nor-malize the noisy nonstandard tokens before applying other NLP techniques. A major challenge facing this task is the system cov-erage, i.e., for any user-created nonstandard term, the system should be able to restore the correct word within its top n output candi-dates. In this paper, we propose a cognitively-driven normalization system that integrates different human perspectives in normalizing the nonstandard tokens, including the en-hanced letter transformation, visual priming, and string/phonetic similarity. The system was evaluated on both word- and message-level using four SMS and Twitter data sets. Results show that our system achieves over 90 % word-coverage across all data sets (a 10 % absolute increase compared to state-of-the-art); the broad word-coverage can also successfully translate into message-level per-formance gain, yielding 6 % absolute increase compared to the best prior approach. 1