RAT - Resilient Allreduce Tree for Distributed Machine Learning
Xinchen Wan, Hong Zhang, Hao Wang, Shuihai Hu, Junxue Zhang, Kai Chen · 2020
Parameter/gradient exchange plays an important role in large-scale distributed machine learning (DML). However, prior solutions such as parameter server (PS) or ring-allreduce (Ring) fall short since they are not resilient to issues or uncertainties like oversubscription, congestion or failures that may occur in datacenter networks (DCN).