A framework for the development of globally convergent adaptive learning rate algorithms
George D. Magoulas, Vassilis P. Plagianakos, George Androulakis, Michael N. Vrahatis, Greece Greece · 1999
: - In this paper we propose a framework for developing globally convergent batch training algorithms with adaptive learning rate. The proposed framework provides conditions under which global convergence is guaranteed for adaptive learning rate training algorithms. To this end, the learning rate is appropriately tuned along the given descent direction. Providing conditions regarding the search direction and the corresponding stepsize length this framework can also guarantee global convergence for training algorithms that use a di#erent learning rate for each weight. To illustrate the e#ectiveness of the proposed approach on various training algorithms simulation results are provided. Key-Words: - Global convergence, learning rate adaptation, batch training algorithms, steepest descent, feedforward neural networks. 1.