A Convergent and Dimension-Independent First-Order Algorithm for Min-Max Optimization
Vijay Keswani, Oren Mangoubi, Sushant Sachdeva, Nisheeth K. Vishnoi · arXiv (Cornell University) · 2020
Motivated by the recent work of Mangoubi and Vishnoi (STOC 2021), we propose a variant of the min-max optimization framework where the max-player is constrained to update the maximization variable in a greedy manner until it reaches a *first-order* stationary point. We present an algorithm that provably converges to an approximate local equilibrium for our framework from any initialization and for nonconvex-nonconcave loss functions. Compared to the second-order algorithm of Mangoubi and Vishnoi, whose iteration bound is polynomial in the dimension, our algorithm is first-order and its iteration bound is independent of dimension. We empirically evaluate our algorithm on challenging nonconvex-nonconcave test-functions and loss functions that arise in GAN training. Our algorithm converges on these test functions and, when used to train GANs on synthetic and real-world datasets, trains stably and avoids mode collapse.