Using Bayesian Optimization to Improve Hyperparameter Search in TPOT
Angus Kenny, Tapabrata Ray, Steffen Limmer, Hemant Kumar Singh, Tobias Rodemann, Markus Olhofer · Proceedings of the Genetic and Evolutionary Computation Conference · 2024
Automated machine learning (AutoML) has emerged as a pivotal tool for applying machine learning (ML) models to real-world problems. Tree-based pipeline optimization tool (TPOT) is an AutoML framework known for effectively solving complex tasks. TPOT's search involves two fundamental objectives: finding optimal pipeline structures (i.e., combinations of ML operators) and identifying suitable hyperparameters for these structures. While its use of genetic programming enables TPOT to excel in structural search, its hyperparameter search, involving discretization and random selection from extensive potential value ranges, can be computationally inefficient. This paper presents a novel methodology that heavily restricts the initial hyperparameter search space, directing TPOT's focus towards structural exploration. As the search evolves, Bayesian optimization (BO) is used to refine the hyperparameter space based on data from previous pipeline evaluations. This method leads to a more targeted search, crucial in situations with limited computational resources. Two variants of this approach are proposed and compared with standard TPOT across six datasets, with up to 20 features and 20,000 samples. The results show the proposed method is competitive with canonical TPOT, and outperforms it in some cases. The study also provides new insights into the dynamics of pipeline structure and hyperparameter search within TPOT.