A scalable and topology configurable protocol for distributed parameter synchronization
Minjie Wang, Hucheng Zhou, Minyi Guo, Zheng Zhang · 2014
This paper addresses the problem of model synchronization in data-parallelism of deep-learning systems. In such systems, workers on different machines continuously update their local copies of the model, and the updates need to be merged so that the copies are roughly consistent to each other. In modern implementations using GPUs, workers generate very high updates, posing significant scalability challenges.