Stochastic Levenberg-Marquardt for Solving Optimization Problems on Hardware Accelerators
Yuxi Hong, Houcine Bergou, Nicolas Doucet, Hao Zhang, Jesse Cranney, Hatem Ltaief, Damien Gratadour, François Rigaut, David Elliot Keyes · King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) · 2020
We present a new Stochastic Levenberg-Marquardt (SLM) algorithm for efficiently solving large-scale nonlinear least-squares optimization problems. The SLM method incorporates stochasticity into the traditional Levenberg-Marquardt (LM) method. While the traditional LM operates on the full objective function, SLM randomly evaluates part of the objective to compute the corresponding derivatives and function values. Hence, SLM reduces the algorithmic complexity per iteration and speeds up the overall time to solution, while maintaining the numerical robustness of second-order methods. We assess the SLM method on standard datasets from LIBSVM, as well as on a large-scale optimization problem found in ground-based astronomy applications, and in particular adaptive optics systems of the next generation of instruments for the European Very Large and Extremely Large Telescopes. We implement SLM and deploy it on a shared-memory system equipped with multiple GPU hardware accelerators. We demonstrate the performance superiority of the SLM method over not only the traditional LM algorithm but also the state-of-the-art first-order methods. SLM finishes optimization process in less than 1 second on large datasets from the adaptive optics application, where LM and other methods require more than a few minutes. This enables to identify of the system parameters (e.g., atmospheric turbulence and wind speed) and to capture their evolution required during a night of observations with a close to real-time throughput.