ADINE: An Adaptive Momentum Method for Stochastic Gradient Descent
Vishwak Srinivasan, Adepu Ravi Sankar, Vineeth Nallure Balasubramanian · arXiv (Cornell University) · 2017
Two major momentum-based techniques that have achieved tremendous success in optimization are Polyak's heavy ball method and Nesterov's accelerated gradient. A crucial step in all momentum-based methods is the choice of the momentum parameter $m$ which is always suggested to be set to less than $1$. Although the choice of $m 1$), evaluate our proposed algorithm on deep neural networks and show that $\textit{ADINE}$ helps the learning algorithm to converge much faster without compromising on the generalization error.