Path Length Bounds for Gradient Descent and Flow
Chirag Gupta, Sivaraman Balakrishnan, Aaditya Ramdas · arXiv (Cornell University) · 2019
We derive bounds on the path length $ζ$ of gradient descent (GD) and gradient flow (GF) curves for various classes of smooth convex and nonconvex functions. Among other results, we prove that: (a) if the iterates are linearly convergent with factor $(1-c)$, then $ζ$ is at most $\mathcal{O}(1/c)$; (b) under the Polyak-Kurdyka-Lojasiewicz (PKL) condition, $ζ$ is at most $\mathcal{O}(\sqrtκ)$, where $κ$ is the condition number, and at least $\widetildeΩ(\sqrt{d} \wedge κ^{1/4})$; (c) for quadratics, $ζ$ is $Θ(\min\{\sqrt{d},\sqrt{\log κ}\})$ and in some cases can be independent of $κ$; (d) assuming just convexity, $ζ$ can be at most $2^{4d\log d}$; (e) for separable quasiconvex functions, $ζ$ is $Θ(\sqrt{d})$. Thus, we advance current understanding of the properties of GD and GF curves beyond rates of convergence. We expect our techniques to facilitate future studies for other algorithms.