Stability of steepest descent with momentum for quadratic functions
Manabu Torii, Martin Hagan · IEEE Transactions on Neural Networks · 2002
This paper analyzes the effect of momentum on steepest descent training for quadratic performance functions. We demonstrate that there always exists a momentum coefficient that will stabilize the steepest descent algorithm, regardless of the value of the learning rate. We also demonstrate how the value of the momentum coefficient changes the convergence properties of the algorithm.