A note on activation function in multilayer feedforward learning
Joarder Kamruzzaman, Syed Mahfuzul Aziz · 2003
Multilayer feedforward network trained by backpropagation algorithm suffers from slow learning speed. One of the reasons of slow convergence is the diminishing value of the derivative of the commonly used activation functions as the nodes approaches saturated values. In this paper, we present a new activation function to accelerate backpropagation learning. A comparison among the commonly used activation functions, recently proposed logarithmic function and the proposed activation function shows accelerated convergence with the proposed one. This activation function can be used in conjunction with other techniques to further accelerate the learning speed or reduce the chance of being trapped in local minima. Simulation using this activation function shows improvement in the learning speed compared with other commonly used functions and the new activation function proposed by Bilski (2000). This function may also be used in other multilayer feedforward training algorithms.