Comprehensive Analysis of Random Forest and XGBoost Performance with SMOTE, ADASYN, and GNUS Upsampling Under Varying Imbalance Levels
Mehdi Imani, Ali Beikmohammadi, Hamid Reza Arabnia · Preprints.org · 2025
This study examines the efficacy of Random Forest and XGBoost classifiers in conjunction with three upsampling techniques—SMOTE, ADASYN, and Gaussian Noise Up-Sampling (GNUS)—across datasets with varying class imbalance levels, ranging from moderate (15% churn) to extreme (1% churn). Employing metrics such as F1-Score, ROC AUC, PR AUC, Matthews Correlation Coefficient (MCC), and Cohen’s Kappa, the research offers a comprehensive evaluation of classifier performance under different imbalance scenarios, with a focus on applications in the telecommunications domain. The findings highlight the consistent superiority of XGBoost paired with SMOTE, achieving the highest F1-Score and robust performance across all levels of class imbalance, including extreme cases. SMOTE proved to be the most effective upsampling method, particularly with XGBoost, enhancing learning from minority class instances. Random Forest, while adequate for moderate imbalances, struggled under severe imbalance, regardless of the upsampling method. ADASYN showed moderate effectiveness with XGBoost but underperformed with Random Forest, whereas GNUS demonstrated inconsistent results, lagging behind SMOTE and ADASYN. The study emphasizes the critical influence of data imbalance on model performance, revealing that metrics such as MCC, Kappa, and F1-Scores fluctuate significantly with increasing imbalance, while ROC AUC and PR AUC remain stable. Ultimately, XGBoost combined with SMOTE emerged as the most robust and effective strategy for addressing extreme class imbalance.