Accelerating training of DNN in distributed machine learning system with shared memory
Eunji Lim, Shinyoung Ahn, Wan Choi · 2017
In distributed DNN training, the speed of reading and updating model parameters greatly affects model training time. In this paper we investigate the performance of deep neural network training with parameter sharing based on shared memory for distributed machine learning. We propose a shared memory-based modification of the deep learning framework. In our framework, remote shared memory is used to maintain global shared parameters of parallel deep learning workers. Our framework can accelerate training of DNN by speeding up the parameter sharing in every training iteration in distributed model training. We evaluated our proposed framework by training the three different deep learning model. The experiment results show that our framework improves training time for deep learning models in distributed system.