Theory and Implementation of the Savvy Ball Method with application to machine learning
Manosalvas Holguín, Peter David Sly · 2020
Recently, computer scientists have considered the use of second order differential equa- tions [1] in order to provide a dynamic search trajectory to train neural networks [2], [3]. They are based on the heavy ball method of B.T. Polyak [4]. Previous research focused on using Polyak’s method in order to speed up the convergence rate to a local minimizer compared to simple steepest descent. We focus here on the glabal optimization aspect and weaken the requirement on the objective to Lipschitz continuity instead of twice continuous differentiability [2]. We analyze theoretically the non-smooth but convex case where the ODE generalizes to an Ordinary Differential Inclusion (ODI). We show numerical results for implementation of what we call Savvy Ball method which was referred to as TOAST in [5], using the parallel programming environment OpenMP.