Mutual Information Reduction Techniques and its Applications in Feature Engineering
Ruixin Chen, David Li · 2025
Feature engineering is a vital aspect of machine learning model development, as it involves selecting and transforming relevant data features to improve predictive accuracy. Traditional feature selection methods focus on maximizing mutual information (MI) between features and the target variable. In contrast, this paper introduces novel mutual information reduction techniques aimed at minimizing redundant information between features. By reducing mutual information among features, these methods enhance feature selection and creation, allowing machine learning models to benefit from less correlated and more informative variables. We provide examples and an implementation to demonstrate how mutual information reduction improves model performance, particularly in classification tasks. Additionally, the integration of Weight of Evidence (WOE) transformation further boosts predictive power by capturing the unique information from each feature.