NCSU-SAS-Ning: Candidate Generation and Feature Engineering for Supervised Lexical Normalization
Jin Ning · 2015
User generated content often contains non-standard words that hinder effective automatic text processing.In this paper, we present a system we developed to perform lexical normalization for English Twitter text.It first generates candidates based on past knowledge and a novel string similarity measurement and then selects a candidate using features learned from training data.The system has a constrained mode and an unconstrained mode.The constrained mode participated in the W-NUT noisy English text normalization competition (Baldwin et al., 2015) and achieved the best F1 score.