Mutual Information: an Adequate Tool for Feature Selection
Benoît Frénay, Gauthier Doquire, Michel Verleysen · Digital Access to Libraries (Université catholique de Louvain (UCL), l'Université de Namur (UNamur) and the Université Saint-Louis (USL-B)) · 2013
In the field of machine learning, mutual information (MI) has been widely used as a multivariate criterion of nonlinear feature relevance (Kojadinovic, 2005; Rossi et al., 2006; Doquire & Verleysen, 2011). Indeed, it is wellknown in information theory that I(X ;Y ) measures the reduction of uncertainty about a target Y when a set of features X are observed. However, other criteria are commonly used for classification and regression to assess the quality of models, like e.g. accuracy or mean square error (MSE). This presentation reviews several works (Frenay et al., 2012b; Frenay et al., 2013; Frenay et al., 2012a; Doquire et al., 2013) which address the relationships between MI and these criteria in a feature selection context.