On the line-search gradient methods for stochastic optimization
Darina Dvinskikh, Aleksandr Ogaltsov, Alexander Vladimirovich Gasnikov, Pavel Dvurechensky, Vladimir Grigor'evich Spokoiny · IFAC-PapersOnLine · 2020
We consider several line-search based gradient methods for stochastic optimization: a gradient and accelerated gradient methods for convex optimization and gradient method for non-convex optimization. The methods simultaneously adapt to the unknown Lipschitz constant of the gradient and variance of the stochastic approximation for the gradient. The focus of this paper is to numerically compare such methods with state-of-the-art adaptive methods which are based on a different idea of taking norm of the stochastic gradient to define the stepsize, e.g., AdaGrad and Adam.