Preliminary Performance Analysis of Distributed DNN Training with Relaxed Synchronization
Koichi Shirahata, Amir Haderbache, Naoto Fukumoto, Kohta Nakashima · IEICE Transactions on Electronics · 2020
Scalability of distributed DNN training can be limited by slowdown of specific processes due to unexpected hardware failures. We propose a dynamic process exclusion technique so that training throughput is maximized. Our evaluation using 32 processes with ResNet-50 shows that our proposed technique reduces slowdown by 12.5% to 50% without accuracy loss through excluding the slow processes.