Aspects of generative and discriminative classifiers
Jing‐Hao Xue · Enlighten: Theses (The University of Glasgow) · 2008
In recent years, under the new terminology of generative and discriminative classifiers, research interest in classical statistical approaches to discriminant analysis has re-emerged in the machine learning community. In discriminant analysis, observations with features x measured are classified into classes labelled by a categorical variable y. Generative classifiers, also termed the sampling paradigm, such as normal-based discriminant analysis and the naïve Bayes classifier, model the joint distribution p(x,y) of the measured features x and the class labels y factorised in the form p(x|y)p(y), where p(x|y) is a data-generating process (DGP), and learn the model parameters through maximisation of the likelihood with respect to p(x|y)p(y). Discriminative classifiers, also termed the diagnostic paradigm, such as logistic regression, model the conditional distribution p(y|x) of the class labels given the features, and learn the model parameters through maximising the conditional likelihood based on p(y|x). In order to exploit the best of both worlds, it is necessary to first compare generative and discriminative classifiers and then combine them. In this thesis, we first performed some empirical and simulation studies to provide extension of and make comments on a highly-cited report (Ng and Jordan, 2001), which compared the naïve Bayes classifier or normal-based linear