Gradient Descent: Second Order Momentum and Saturating Error
Barak A. Pearlmutter · Maynooth University ePrints and eTheses Archive (Maynooth University) · 1991
Batch gradient descent, \\Deltaw(t) = \\GammajdE=dw(t), converges to a minimum of quadratic form with a time constant no better than 1 4 max= min where min and max are the minimum and maximum eigenvalues of the Hessian matrix of E with respect to w. It was recently shown that adding a momentum term \\Deltaw(t) = \\GammajdE=dw(t) + ff\\Deltaw(t \\Gamma 1) improves this to 1 4 p max = min , although only in the batch case. Here we show that secondorder momentum, \\Deltaw(t) = \\GammajdE=dw(t) + ff\\Deltaw(t \\Gamma 1) + fi \\Deltaw(t \\Gamma 2), can lower this no further. We then regard gradient descent with momentum as a dynamic system and explore a nonquadratic error surface, showing that saturation of the error accounts for a variety of effects observed in simulations and justifies some popular heuristics. 1 INTRODUCTION Gradient descent is the bread-and-butter optimization technique in neural networks. Some people build special purpose hardware to accelerate gradient descent optimization...