Adaptive Parser-Centric Text Normalization
Congle Zhang, Tyler Baldwin, Howard Ho, Benny Kimelfeld, Yunyao Li · 2013
Text normalization is an important first step towards enabling many Natural Lan-guage Processing (NLP) tasks over infor-mal text. While many of these tasks, such as parsing, perform the best over fully grammatically correct text, most existing text normalization approaches narrowly define the task in the word-to-word sense; that is, the task is seen as that of mapping all out-of-vocabulary non-standard words to their in-vocabulary standard forms. In this paper, we take a parser-centric view of normalization that aims to convert raw informal text into grammatically correct text. To understand the real effect of nor-malization on the parser, we tie normal-ization performance directly to parser per-formance. Additionally, we design a cus-tomizable framework to address the often overlooked concept of domain adaptabil-ity, and illustrate that the system allows for transfer to new domains with a minimal amount of data and effort. Our experimen-tal study over datasets from three domains demonstrates that our approach outper-forms not only the state-of-the-art word-to-word normalization techniques, but also manual word-to-word annotations. 1