Comparative Study of Surrogate Techniques for CNN Hyperparameter Optimization
Universiti Teknologi PETRONAS, Seri Iskandar, Perak, Malaysia, Nurshazlyn Mohd Aszemi, Nordin Zakaria, P.D.D. Dominic · Soft Computing Research Society eBooks · 2021
Optimizing hyper parameters in Convolutional Neural networks is a tedious process for many researchers and practitioners. It requires a high degree of expertise or experience to optimize the hyper parameters, and manual optimization is likely to be biased. To date, methods or approaches to automate hyper parameter optimization include grid search, random search, and Genetic Algorithms (GAs). However, evaluating large number of sample points in the hyperparameter configuration space, as is typically required by these methods, is a computationally expensive process. Hence, the objective of this paper is to explore regression as a surrogate technique in CNN hyperparameter optimization. Performance in terms of accuracy, error rate, training time and coefficient of determination (R2) are evaluated and recorded. Although there is no significant performance difference between the resulting optimized Deep Learning and state-of-the-art on CIFAR-10 datasets, using regression as a surrogate technique for CNN hyperparameter optimization contributes to minimizing the time taken for the optimization process, a benefit which has not been fully explored in the literature to the best of the author's knowledge.