Exploring the Statistical Properties and Developing a Non-Linear Activation Function
Ochin Sharma · 2022 International Conference on Automation, Computing and Renewable Systems (ICACRS) · 2022
To improve the performance of deep learning, it is required to comprehend the benefits and drawbacks of activation functions as they engage in a significant part in the neural networks. Broadly, activation functions can be divided into linear and nonlinear functions. The well-known activation functions such as relu, LRelu, swish, selu, elu, softmax also lacks in the mathematical properties as a function. As an alternative to the well-known sigmoid function, typical types of nonlinear activation functions will be introduced in this study, followed by an evaluation of their properties. Additionally, deeper neural networks will be examined because they have a beneficial impact on results compared to shallower networks. The effect of selecting weights from Gaussian and uniform distributions will be examined, paying particular emphasis to how the quantity of incoming and outgoing connections to a node affects the overall network. They also strictly depend on the weight initialization.