Communication Scheduling Optimization for Distributed Deep Learning Systems
Ching-Yuan Tsai, Ching‐Chi Lin, Pangfeng Liu, Jan‐Jan Wu · 2018
Deep learning is an increasingly important technique that can solve complex problems. Due to the growth of data and model complexity, large-scale deep learning has became an important issue. Distributed deep learning is an efficient way to address these complexity issues in training a huge model. However, in a distributed environment network bandwidth becomes a performance bottleneck for deep learning. We propose various optimizations in reducing network usage by scheduling network request events properly, so as to reduce the total training time. These scheduling optimization only requires software innovation and without the need to upgrade physical network bandwidth, thus is economically competitive. The experiments indicate that our scheduler achieves up to 25 % speedup over traditional schedulers.