Adaptive Gradient Descent for Convex and Non-Convex Stochastic Optimization
Darina Dvinskikh, Aleksandr Ogaltsov, Alexander Vladimirovich Gasnikov, Pavel Dvurechensky, Tyurin, Alexander, Vladimir Grigor'evich Spokoiny · arXiv (Cornell University) · 2019
In this paper we propose several adaptive gradient methods for stochastic optimization. Unlike AdaGrad-type of methods, our algorithms are based on Armijo-type line search and they simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. We consider an accelerated and non-accelerated gradient descent for convex problems and gradient descent for non-convex problems. In the experiments we demonstrate superiority of our methods to existing adaptive methods, e.g. AdaGrad and Adam.