Towards Robust Cluster-Based Hyperparameter Optimization
Leonardo Izaú, Mariana Fortes, Vitor Ribeiro, Celso Marques, Carla Silva Oliveira, Eduardo Augusto Duque Bezerra, Fabio Andre Machado Porto, Rebecca Pontes Salles, Eduardo Ogasawara · 2022
Hyperparameter optimization is a fundamental step in machine learning pipelines since it can influence the predictive performance of the resulting models. However, the setup generally selected by classical hyperparameter optimization based on minimizing an objective function may not be robust to overfitting. This work proposes CHyper, a novel clustering-based approach to hyperparameter selection. CHyper derives a candidate cluster of close or similar hyperparameters with low prediction errors in the validation dataset. Hyperparameters chosen are likely to produce models that generalize the inherent behavior of the data. CHyper was evaluated with two different clustering techniques, namely k-means and spectral clustering, in the context of time series prediction of annual fertilizer consumption. Complementary to minimizing an objective function, cluster-based hyperparameter selection achieved robustness to negative overfitting effects and contributed to lowering a generalization error.