Asynchronous SGD for DNN training on Shared-memory Parallel Architectures
Florent Lopez, Edmond Chow, Stanimire Z. Tomov, Jack J. Dongarra · 2020
We present a parallel asynchronous Stochastic Gradient Descent algorithm for shared memory architectures. Different from previous asynchronous algorithms, we consider the case where the gradient updates are not particularly sparse. In the context of the MagmaDNN framework, we compare the parallel efficiency of the asynchronous implementation with that of the traditional synchronous implementation. Tests are performed for training deep neural networks on multicore CPUs and GPU devices.