Convergence analysis of feedforward neural networks with backpropagation
Avazjon Rakhimovich Marakhimov, Kabul Kadirbergenovich Khudaybergenov · Bulletin of National University of Uzbekistan Mathematics and Natural Sciences · 2019
In this paper, convergence analysis properties of online gradient training for backpropagation algorithm for feedforward neural networks with a two hidden layer is studied. We assume that in every training cycle, every training pattern in the training dataset is fed in a stochastic form to the feedforward multilayer neural network exactly once. In this study, we give a weak and strong convergence properties for the training approaches, indicating that the gradient of the error function goes to zero and the weights goes to a fixed point value, respectively. First, we give convergence result for completely stochastic order approach and then follows for special stochastic order approach. The conditions on the activation function of the network and the training rate to guarantee the convergence are relaxed compared with the existing results. Convergence properties in the current paper are studied for sigmoidal activation function type, however this results are also valid for other type of activation functions.