A Comparison of Weight Initializers in Deep Learning
Akibu Mahmoud Abdullahi, Nyi Nyi Nay Naing, Md Shakil Hossain, Sai Ahkar Htet, Shahrinaz Ismail, Syaza Lyana Mahadzir · 2023
Deep learning has revolutionized the field of artificial intelligence and machine learning by enabling the development of complex neural network architectures capable of solving a wide range of real-world problems. Weight initialization in deep learning is a critical phase in the training process that involves assigning initial values to neural network parameters. The challenge at hand is finding the correct balance in initializing weights - not setting them too high or too low - to promote effective training while avoiding complications like disappearing or ballooning gradients. This paper aims to compare the weight initializers in deep learning algorithms provided by Keras Library. The findings indicate that the weight initialization approach used has a considerable impact on the model’s performance. Simpler approaches, such as Zeros and Ones, perform poorly in this case, resulting in slower convergence and less accurate models. Random initializations are superior for learning, with RandomNormal surpassing RandomUniform. Variance and TruncatedNormalScaling approaches show a better combination of exploration and convergence.