From Nonlinear Optimization to Neural Network Training
Jenq–Neng Hwang, Paul S. Lewis · 2005
In this paper we examine the application of second order learning algorithms, based on Newton’s method, to the training of multilayer perceptrons. These methods accelerate learning by estimating the local curvature of the training error surface. Basic second order algorithms require O(N2) operations per update, where N is the number of parameters in the network. For block training of a network, this is in balance with the O(N2) operations required to compute the complete training set error and its derivative.