Hybrid tuning of activation functions in feedforward neural networks
Leandro Nunes de Castro, Luis A. Ramirez, Fernando A. C. Gomide, Fernando José Von Zuben · 2003
Tuning procedures for activation functions significantly increases the flexibility and the nonlinear approximation capability of feedforward neural networks in supervised learning tasks. As a consequence, the learning process presents a better performance, with the final state of the neural network being kept away from undesired saturation regions. Based on a hybrid architecture combining a gradient strategy with a fuzzy decision model, an auto-tuning algorithm is derived to adjust additional parameters associated with the activation functions. The other conventional parameters, the connection weights between layers, are adjusted using a powerful second-order approach based on a conjugate gradient algorithm. To demonstrate the performance of the proposed method we compare this technique with the standard algorithm and with an auto-tuning strategy based solely on the gradient descent method. The three algorithm are applied to several artificial and real world benchmarks.