STUDIES ON THE GLOBAL CONVERGENCE OF GRADIENT DESCENT FOR OVER-PARAMETERIZED MODELS USING OPTIMAL TRANSPORT: COMPUTATIONAL OPTIMAL TRANSPORT PROJECT (MVA)
Mengda Li · HAL (Le Centre pour la Communication Scientifique Directe) · 2020
This is the project report of MVA Master 1 course Computational Optimal Transport on the studies of [1]: On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport and [2]: Sparse Optimization on Measures with Over-parameterized Gradient Descen writen by Lénaïc Chizat and Francis Bach. The results of [1] are qualitative while the [2]'s are more quantitative. The [1] aims to explain when and why the non-convex particle gradient descent finds global minima by studying the many-particle limit of the gradient flow. The [2] studies almost the same optimization problem as [1] with a compact d-dimensional Riemannian mani-fold parameter space. One of the biggest relevant interest of [1] is to give a convergence analysis of the optimization problem of neural network (at least the 2-layer one). The [1] does not provide any new algorithm while the [2] propose a Conic Particle Gradient Descent algorithm with retractions on a Riemannian manifold. The contribution of [1] is on the theoretical side: it gives some qualitative convergence theorems on Wasserstein gradient flow. As the particle gradient flow is a particular case of the Wasserstein one and the gradient descent method is only a discretization of the differential equation of the particle gradient flow with fixed particle number, this contribution could be served in the convergence analysis of more complicated deep neural network. On the practical side, the [1] only shows some simple numerical illustrations, for example the convergence of gradient descent of a 2-layer ReLU network with different numbers of particles. In this project, I focus on [1].