Learning data set similarities for hyperparameter optimization initializations
Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme · 2015
Abstract. Current research has introduced new automatic hyperpa-rameter optimization strategies that are able to accelerate this opti-mization process and outperform manual and grid or random search in terms of time and prediction accuracy. Currently, meta-learning methods that transfer knowledge from previous experiments to a new experiment arouse particular interest among researchers because it allows to improve the hyperparameter optimization. In this work we further improve the initialization techniques for sequen-tial model-based optimization, the current state of the art hyperparame-ter optimization framework. Instead of using a static similarity prediction between data sets, we use the few evaluations on the new data sets to cre-ate new features. These features allow a better prediction of the data set similarity. Furthermore, we propose a technique that is inspired by active learning. In contrast to the current state of the art, it does not greedily choose the best hyperparameter configuration but considers that a time budget is available. Therefore, the first evaluations on the new data set are used for learning a better prediction function for predicting the simi-larity between data sets such that we are able to profit from this in future evaluations. We empirically compare the distance function by applying it in the scenario of the initialization of SMBO by meta-learning. Our two proposed approaches are compared against three competitor meth-ods on one meta-data set with respect to the average rank between these methods and show that they are able to outperform them. 1