Using Maximum Entropy Models to Discriminate between Similar Languages and Varieties
Jordi Porta, José Luis Sancho · 2014
DSLRAE is a hierarchical classifier for similar written languages and varieties based on maximum-entropy (maxent) classifiers.In the first level, the text is classified into a language group using a simple token-based maxent classifier.At the second level, a group-specific maxent classifier is applied to classify the text as one of the languages or varieties within the previously identified group.For each group of languages, the classifier uses a different kind and combination of knowledge-poor features: token or character n-grams and 'white lists' of tokens.Features were selected according to the results of applying ten-fold cross-validation over the training dataset.The system presented in this article 1 has been ranked second in the Discriminating Similar Language (DSL) shared task co-located within the VarDial Workshop at COLING 2014 (Zampieri et al., 2014).