Comparative Analysis of Hyperparameter Tuning Methods in Classification Models For Ensemble Learning
Hamzah Dabool, Hany Alashwal, Hamda Alnuaimi, Asma Alhouqani, Shaikha Alkaabi, Amal Al Ahbabi · 2024
Hyperparameter tuning plays a critical role in optimizing machine learning models, directly impacting their accuracy and generalization capabilities. In this paper, we implement and compare four prominent hyperparameter tuning algorithms: Grid Search, Random Search, Bayesian Optimization, and Genetic Algorithm. Our goal is to evaluate these methods on multiclass classification task, assessing them based on tuning time, computational complexity, accuracy score, and ease of use. Through an extensive experimental analysis, we identify the strengths and limitations of each approach, providing insights into their ideal use cases. The results reveal trade-offs between exhaustive search methods like Grid Search, which offer higher accuracy at the cost of time, and more efficient alternatives like Random Search and Bayesian Optimization, which balance exploration and exploitation. Genetic Algorithms, while less commonly used, show potential in discovering global optima.