An Asynchronous Distributed Training Algorithm Based on Gossip
Jia Zhou, Jun Tu, Donglin Ren · 2022
Existing decentralized deep learning algorithms have been widely used in deep learning. But since they are synchronous, the communication waiting time inter-node in the cluster has become longer. A novel asynchronous decentralized distributed deep learning algorithm Gossip Ring SGD (GR- SGD) is proposed in this paper. This paper uses the ImageNet data set and the ResNet model to verify the feasibility of the algorithm and com- pared with Ring AllReduce and D-PSGD, it is proved that the algorithm can complete training in a shorter time.