On Guo and Nixon's Criterion for Feature Subset Selection: Assumptions, Implications, and Alternative Options
Kiran S. Balagani, Vir V. Phoha, S. S. Iyengar, N. Balakrishnan · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · 2010
Guo and Nixon proposed a feature selection method based on maximizingI(x;Y), the multidimensional mutual information between feature vectorxand class variableY. Because computingI(x;Y) can be difficult in practice, Guo and Nixon proposed an approximation ofI(x;Y) as the criterion for feature selection. We show that Guo and Nixon's criterion originates from approximating the joint probability distributions inI(x;Y) by second-order product distributions. We remark on the limitations of the approximation and discuss computationally attractive alternatives to computeI(x;Y) .