Enhancing Regression Accuracy and Reliability with Ensemble Learning and Iterative Hyperparameter Optimization
Thi Giang Thanh Tran, Thanh Tran, Thanh Tran, Thanh Tran · Engineering Technology & Applied Science Research · 2025
Ensemble learning models such as XGBoost, LightGBM, and CatBoost are gaining popularity for regression predictions. As their accuracy significantly depends on hyperparameters, identifying the optimal hyperparameters plays a crucial role. This study proposes a new approach by integrating ensemble learning models with hyperparameter optimization algorithms to assess their effectiveness compared to traditional machine learning models such as RF, SVR, and KNN. This study also conducted multiple repetitions of the hyperparameter optimization process on the Boston Housing Price and Bike Sharing Demand datasets, supported by statistical analyses of the results to enhance the reliability of the findings compared to a single run. The results demonstrate that applying ensemble learning models helps reduce prediction errors, increasing the models' accuracy compared to traditional ML models. Furthermore, comparisons indicate that repeating iterations n times improves the reliability of the results.