Optimizing deep neural networks hyperparameter positions and values
Ahmed A. Akl, Ibrahim El-Henawy, Ahmad Salah, Kenli Li · Journal of Intelligent & Fuzzy Systems · 2019
Hyperparameter optimization is a crucial step in the implementation of any machine learning model. This optimization process includes regularly modifying the hyperparameter values of the model in order to minimize the testing error. A deep neural learning model hyperparameter optimization process includes optimizing both the model parameters and architecture. Optimizing a model’s parameters involves deciding the values of parameters, such as learning rate and batch size. Optimizing architectural hyperparameters includes deciding the shape of the deep neural learning model, i.e. , the number of layers of individual types and the number of neurons in a certain layer. The state-of-the-art hyperparameter optimization methods don’t optimize the position of the hyperparameter within the model architecture. In this work, we study the effect of changing a hyperparameter within the deep learning model architecture. Thus, we propose an arch itectural pos ition opt imization ( ArchPosOpt ) method for model architectural hyperparameter optimization. ArchPosOpt extends three different hyperparameter optimization techniques, namely grid search, random search, and Tree-structured Parzen Estimator (TPE), to include a new dimension of hyperparameter optimization problem – the hyperparameter position. We show through a set of experiments that the position of the hyperparameters does matter for model performance as well as the hyperparameter values.