Faster Hyperparameter Optimization via Finding Minimal Regions in Random Forest Regressor
Viacheslav Shalamov, Valeria Efimova, Andrey A. Filchenkov · Procedia Computer Science · 2022
In optimisation problems, it is often more convenient and efficient to approximate a target function with another function, which would be much easier to compute or analyse. Such approximations are called Surrogate Functions. A surrogate function can also be a fairly complex object itself. For example, a Random Forest Regressor (RFR) can be used to approximate target function in Hyperparameter Optimisation processes. Hyperparameter vectors define the feature space, and target algorithm performance is considered the target function. Thus, RFR trained on a set of examples, can approximate algorithm performance, using hyperpa-rameter vectors. However, it is hard to fnd its minima. Thus, “black-box” optimization algorithms are applied for optimization purposes, such as random or local searches. In this paper, we propose a method ShortestForestSearch to search RFR minimum points using branch and bound approach. We conducted several experiments for comparing the algorithm proposed with näıve approaches. ShortestForestSearch demonstrated significant performance improvements and the potential to improve other algorithms which use Random Forest internally as a surrogate function. Based on this, we propose ShortestForestSMAC, a new algorithm for hyperparameter optimization. It is based on the well-known SMAC algorithm and implements idea of ShortestForestSearch for selecting best new candidates on each optimization iteration. Experimental evaluation shows that ShortestForestSMAC outperforms SMAC by achieving similar results in considerably shorter time span.