Communication-Efficient Decentralized Zeroth-order Method on Heterogeneous Data
Zan Li, Li Chen · 2021 13th International Conference on Wireless Communications and Signal Processing (WCSP) · 2021
Decentralized federated learning is an communication efficient method to learn from large-scale distributed dataset. Most existing FL algorithms are gradient-based and assume knowledge of first-order information of the objective function in client nodes. However, there are situations where the information of gradient is difficult to touch or compute accurately. In this paper, we consider a local zeroth-order algorithm based on biased stochastic zeroth-order update in decentralized federated learning setting. We prove concise convergence rates on strongly convex problems and show that it matches the rate of decentralized local stochastic gradient descent (local SGD) up to a factor proportional to the dimension of the parameter. The number of communication rounds can be reduced by$O(\sqrt{T}/N)$times. Simulation results confirm that our algorithm is globally convergent and further reveal communication complexity improvement over traditional distributed zeroth-order methods.