Semi-Supervised Text Classification Using EM
Kamal Nigam, Andrew McCallum, Tom M. Mitchell · The MIT Press eBooks · 2006
For several decades, statisticians have advocated using a combination of labeled and unlabeled data to train classifiers by estimating parameters of a generative model through iterative expectation-maximization (EM) techniques.This chapter explores the effectiveness of this approach when applied to the domain of text classification.Text documents are represented here with a bag-of-words model, which leads to a generative classification model based on a mixture of multinomials.This model is an extremely simplistic representation of the complexities of written text.This chapter explains and illustrates three key points about semi-supervised learning for text classification with generative models.First, despite the simplistic representation, some text domains have a high positive correlation between generative model probability and classification accuracy.In these domains, a straightforward application of EM with the naive Bayes text model works well.Second, some text domains do not have this correlation.Here we can adopt a more expressive and appropriate generative model that does have a positive correlation.In these domains, semi-supervised learning again improves classification accuracy.Finally, EM suffers from the problem of local maxima, especially in high-dimension domains such as text classification.We demonstrate that deterministic annealing, a variant of EM, can help overcome the problem of local maxima and increase classification accuracy further when the generative model is appropriate.