Comparison of Activation Functions in the KarcıFANN Method
Meral Karakurt, Hülya Saygılı, Ali Karcı · 2024
The lack of a certain standard in modeling Artificial Neural Networks (ANNs) sometimes provides successful results while sometimes it causes unsuccessful results depending on the changes in the selected hyper parameters. The optimization method is a hyper parameter that significantly affects the performance of ANNs. If the learning coefficient parameter used in the SGD optimization method, which is a fixed number, is chosen too small, the solution takes a long time; if it is chosen too large, it causes problems such as getting stuck in local minima and moving away from the solution. In the Karcı Fractional Neural Network (KarcıFANN) method proposed in this article, the learning of the network is performed by using the fractional derivative instead of the learning coefficient parameter. Thanks to the KarcıFANN method, the weights are updated with a fractional derivative that changes according to the value of the error function obtained at each iteration and external intervention to the network is minimized. Activation function is another hyperparameter that affects the performance of ANNs. Activation function determines whether the output value of a neuron will be produced or not, that is, whether the neuron will be active or not. In this study, Gina_Prior2 data set was classified using KarcıFANN method using sigmoid, hyperbolic tangent (tanh) and hyperbolic tangent sigmoid (tansig) activation functions and the effects of these activation functions on performance were compared. As a result of the experiments performed on the designed model, it was seen that the most successful activation function was the sigmoid function, which produced accuracy values over 99%.