MSc Project Feature Selection using Information Theoretic Techniques
Adam Pocock · 2008
This document presents a investigation into 3 different areas of feature selection, using information theoretic methods. The first area of research is an investigation into the selection of the first feature in common feature selection algorithms. This step is often overlooked in the construction of feature selection algorithms, with the assumption that the most informative feature is the best first choice. This can be proven to be untrue, and so an investigation into how to select a better suited feature forms the first part of the research. New methods for selecting the first feature are proposed and empirically tested to see if they offer an improvement over the standard method. The second area of research is an investigation into applying the Rényi extension to infor-mation theory to standard feature selection techniques. This requires the development of a Rényi mutual information measure, and two different measures are proposed. The Rényi extension provides a positive real parameter, α, which can be varied. The new Rényi feature selection techniques are empirically tested, varying the measure and value of α used.