A Comparative Study on Automatic Model and Hyper-Parameter Selection in Classifier Ensembles
Antonino Feitosa Neto, João C. Xavier-Júnior, Anne M. P. Canuto, Alexandre Cêsar Muniz de Oliveira · 2019
This work performs an empirical study on Automated Machine Learning (Auto-ML) systems for automatically selecting the best Classifier Ensembles and their hyper-parameter settings or by only selecting the hyper-parameter of a predetermined Classifier Ensemble. In order to perform this analyses, we compared the two selection strategies using the Auto-WEKA system with two bio-inspired algorithms (Genetic and Particle Swarm Optimization), aiming to determine which of these strategies generate more accurate Classifier Ensembles, given a time constraint. We used 15 classification datasets for evaluating the performance of the aforementioned strategies in selecting Classifier Ensembles (e.g., Random Forest, Bagging, Boosting and Vote) and their hyper-parameter. In order to conduct a more robust analysis of the results, we applied statistical tests. Our findings indicate the use of hyper-parameter selection applied to Random Forest might generate more accurate systems compared to model and hyper-parameters selection.