Sep: A New Nonlinear Activation Function for Biomedical Applications and Image Classification
Debasish Saha, Ravi Kumar Jatoth, Y. N. Reddy · 2023
The premise of non-linearity in a neural network is established by an activation function that is crucial for network training and performance evaluation. For many years of theoretical research, lots of activation function have been introduced, but only a certain number of are extensively used in almost all applications including TanH (Tan Hyperbolic), ReLu(Rectified Rectified LinearUnit), Sigmoid, swish, LeakyReLu, Mish. In this research work a nonlinear, nonmonotonic, novel activation Sep is introduced that can be defined as$\mathrm{f}(\theta)=\theta\ ^{*}\sin(\text{sigmoid}(\theta))$. This study demonstrates that the activation function Sep, combined with other common activation functions in neural networks outperforms Relu, Swish, and Mish on various difficult datasets. For example, using a LeNet architecture with Sep to classify the MNIST datasets increased Top-1 test accuracy by 0.44%, 0.22%, and 0.21 % in comparison to the same network with Swish, Mish, and ReLU, respectively.