Aspects of Deep Learning: Hyper-Parameter Tuning, Regularization, and Normalization
Syed Muzamil Basha, Dharmendra Singh Rajput · Intelligent Systems · 2019
Deep learning today is applied to many different application areas and that intuitions about hyper-parameter settings from one application will differ to another application. There is a lot of cross-fertilization among different application domains. To implement more complex models, such as convolutional neural networks or recurring neural networks, is not practical to implement everything yourself from scratch. Fortunately, there are now many good deep learning software frameworks that can help you implement these models. This chapter, focus on learning how to implement a Neural Network. Ranging from aspects like: Hyper-parameter tuning helps the network to train quickly. Setting up user data helps in quickly finding a good high-performance neural network. Analyzing the impact of bias and variance on overall performance. Applying different forms of regularization toward reducing variance of Neural network. Making Optimization algorithm runs quickly with reasonable learning rate using Batch normalization. Additionally, A sense of the typical structure of a TensorFlow program is developed using Python in Jupyter notebook and to start up TensorFlow. The Recommendations made by the research work 172 is as follows: 99.5% training and 0.25% development, 0.25% testing dataset ratio, is preferred in setting up the data. L2 Regularization is used to reduce variance, Adam optimization algorithm to be used update weights of the gradient. Batch normalization makes hyper-parameter search problem much easier, makes the neural network much more robust. So all the modern deep learning programming framework makes it really easy to code up even pretty complex neural networks.