Automatic Tuning of Hyperparameters for Neural Networks in Serverless Cloud
Alex Kaplunovich, Yelena Yesha · 2020
Deep Neural Networks are used to solve the most challenging world problems. In spite of the numerous advancements in the field, most of the models are being tuned manually. Experienced Data Scientists have to manually optimize hyperparameters, such as dropout rate, learning rate or number of neurons for Big Data applications. We have implemented a flexible automatic real-time hyperparameter tuning methodology. It works for arbitrary models written in Python and Keras. We also utilized state of the art Cloud services such as trigger based serverless computing (Lambda), and advanced GPU instances to implement automation, reliability and scalability.The existing tuning libraries, such as hyperopt, Scikit-Optimize or SageMaker, require developers to provide a list of hyperparameters and the range of their values manually. Our novel approach detects potential hyperparameters automatically from the source code, updates the original model to tune the parameters, runs the evaluation in the Cloud on spot instances, finds the optimal hyperparameters, and saves the results in the No-SQL database. The methodology can be applied to numerous Big Data Machine Learning systems.