DWUPP: Dynamic Weight Update through Pipelined Parallel for Distributed Training Model
Yingchi Mao, Zijian Tu, Hua Nie, Ping Ping, Jianxin Huang, Jun Wu · 2022
The pipeline system is available to process multiple training batches simultaneously, thereby allowing forward and backward propagation tasks for each batch to be performed across multi-time units on multi-compute nodes. Different versions of weights were used for calculation throughout the process. However, the weight obsolescence may lead to the instability of model training and the loss of precision. Considering the complexity of the solution to the problem of pipeline parallel model convergence in weight obsolescence, we proposed the dynamic weight update through pipelined parallel for distributed training model (DWUPP) according to the version differences. Based on asynchronous pipeline parallel training, we studied a more accurate weight prediction method to improve the accuracy of the model, and ensure the effectiveness of model training. It can be seen from the experimental results that, compared with the PipeDream method, the accuracy of DWUPP method is increased by 0.27% on average in the verification set, which is superior to that of the other three pipeline parallel training methods. Due to the computational complexity of the weight prediction method, the throughput of the method is slightly lower than that of the PipeDream method and SpecTrain method, but higher than that of the XPipe method. It can effectively accelerate the model training process.