A Log-Linear Model for Unsupervised Text Normalization
Yi Yang, Jacob Eisenstein · 2013
We present a unified unsupervised statistical model for text normalization.The relationship between standard and non-standard tokens is characterized by a log-linear model, permitting arbitrary features.The weights of these features are trained in a maximumlikelihood framework, employing a novel sequential Monte Carlo training algorithm to overcome the large label space, which would be impractical for traditional dynamic programming solutions.This model is implemented in a normalization system called UNLOL, which achieves the best known results on two normalization datasets, outperforming more complex systems.We use the output of UNLOL to automatically normalize a large corpus of social media text, revealing a set of coherent orthographic styles that underlie online language variation.