Hierarchical Character-Word Models for Language Identification
Aaron Jaech, George Mulcaire, Shobhit Hathi, Mari Ostendorf, Noah A. Smith · 2016
Social media messages' brevity and unconventional spelling pose a challenge to language identification.We introduce a hierarchical model that learns character and contextualized word-level representations for language identification.Our method performs well against strong baselines, and can also reveal code-switching.