A Dataset and Classifier for Recognizing Social Media English
Su Lin Blodgett, Johnny Tian-Zheng Wei, Brendan O’Connor · 2017
While language identification works well on standard texts, it performs much worse on social media language, in particular dialectal language-even for English.First, to support work on English language identification, we contribute a new dataset of tweets annotated for English versus non-English, with attention to ambiguity, codeswitching, and automatic generation issues.It is randomly sampled from all public messages, avoiding biases towards preexisting language classifiers.Second, we find that a demographic language modelwhich identifies messages with language similar to that used by several U.S. ethnic populations on Twitter-can be used to improve English language identification performance when combined with a traditional supervised language identifier.It increases recall with almost no loss of precision, including, surprisingly, for English messages written by non-U.S.authors.Our dataset and identifier ensemble are available online. 1