Rational activation functions in neural network with uniform norm based loss function and its application in classification
Communications in Optimization Theory · 2022
In this paper, we demonstrate an application of generalised rational uniform (Chebyshev) approximation to neural networks.In particular, our activation functions are rational functions of degree (1, 1) and the loss function is based on the uniform norm.In this setting, when the coefficients of the rational activation function are fixed, overall optimisation problem of the neural network can be formulated as a generalised rational approximation problem with additional linear constraints.In this case, the weights and the bias of the network are the decision variables.To optimise the decision variables, we suggest using two prominent methods: bisection method and differential correction algorithm.We illustrate the efficiency of this application by performing numerical experiments on classification problems with two classes and report the classification accuracy obtained by the network using the bisection method, differential correction algorithm along with the standard MATLAB toolbox which uses least square loss function.We show that the choice of the uniform norm based loss function with rational activation function leads to a better classification accuracy when the training dataset is either very small or if the classes are imbalanced.