Investigating the Efficacy of a Newly Proposed Activation Function on Deep Neural Networks
Gaurav Kumar Pandey, Sumit Srivastava · 2023
The use of an AF is crucial in DNNs (DNNs) since it serves to incorporate non-linearity into the model. Although numerous AFs have been put forth in the literature, a more potent AF is still required in order to enhance DNN performance. In this research, we suggest a brand-new AF termed “CustomMade” and assess its effectiveness on DNNs. The Tanh function, which is frequently used in DNNs, has been changed to become the CustomMade function. On the CIFAR-10 dataset, we assess the CustomMade function's performance using cutting-edge DNN architectures like ResNet. Our tests demonstrate that, in terms of accuracy and convergence time, the CustomMade function surpasses both cutting-edge AFs and other frequently employed AFs, such as Leaky ReLU and ELU. Additionally, we perform ablation studies to examine the effects of various CustomMade function hyperparameters on DNN performance. Our findings demonstrate that the CustomMade function may deliver consistent performance across various architectures and datasets while being resilient to hyperparameter modification. Overall, our results point to the CustomMade function as a potential replacement for the often employed AFs in DNNs, and they also show that it may result in improved performance and quicker convergence,