Cyclic Coordinate-Update Algorithms for Fixed-Point Problems: Analysis and Applications
Yat Tin Chow, Tianyu Wu, Wotao Yin · SIAM Journal on Scientific Computing · 2017
Many problems reduce to the fixed-point problem of solving $x=T(x)$, where $T$ is a mapping from a Hilbert space to itself. To this problem we apply the coordinate-update algorithms, which update only one or a few components of $x$ at each step. When each step is cheap, these algorithms are faster than the full fixed-point iteration (which updates all the components). In this paper, we focus on cyclic coordinate selection rules, where the ordering of coordinates in each cycle is arbitrary. The corresponding algorithms are fast, but their convergence is unknown in the fixed-point setting. When $T$ is a nonexpansive operator and has a fixed point, we show that the sequence of coordinate-update iterates converges to a fixed point under proper step sizes. This result applies to the primal-dual coordinate-update algorithms, which have wide applications to optimization problems with nonseparable nonsmooth objectives, and/or global linear constraints. Numerically, we apply coordinate-update algorithms with cyclic, shuffled cyclic, and random selection rules to $\ell_1$-robust least squares, total variation minimization, and nonnegative matrix factorization. These algorithms converge much faster than the standard fixed-point iteration. Among the three rules, cyclic and shuffled cyclic rules are overall faster than the random rule.