LDBS-SemiUT: A Comprehensive Training Scheme for Distributed Deep Learning in Heterogeneous Environments
Mingrui Zhu, Zhi Ling, Shengang Deng, Xiaofeng Jiang, Jian Yang · 2024
Distributed deep learning with data parallelism offers promising performance but suffers in heterogeneous environments due to stragglers, where faster workers are forced to wait. To address this, asynchronous approaches improve training efficiency but introduce gradient staleness, compromising model quality. Few studies have considered both training efficiency and model quality during training. In this paper, we design a model update method called Semi-synchronization based on Update Threshold (SemiUT) and a batch size adjustment algorithm called Lyapunov Optimization based Dynamic Batching (LDBS). Subsequently, we combine SemiUT and LDBS into a comprehensive scheme called LDBS-SemiUT. We have conducted extensive evaluations to demonstrate the effectiveness of our scheme. To achieve 65% test accuracy on CIFAR10 with LeNet, LDBS-SemiUT obtains 31.13%, 64.39%, and 30.48% time saving, compared to BSP, ASP and DBS, respectively. Meanwhile, LDBS-SemiUT obtains 0.49%, 0.70% and 2.74% higher final test accuracy than SemiUT, DBS and ASP, respectively.