APRNN: A New Active Propagation Training Algorithm for Distributed RNN
Dejiao Niu, Tianquan Liu, Xia Zheng, Tao Cai, Yawen Liu, Yongzhao Zhan · 2018
The number of hidden neurons in recurrent neural networks (RNNs) greatly affects the model accuracy. When fewer neurons are used, less training time is involved and the accuracy is in a lower degree. Once we increase the number of hidden units, the performance will be encouraged but the training overhead will enhance exponentially. In this work, we propose a novel active propagation training strategy for distributed RNN, which is inspired by the information transmission in biological neuron system. With the aim to improve the training efficiency of RNN, the proposed active propagation avoids the traditional fixed number of neurons in training. Instead, when training the distributed RNN, the autonomous neuron dynamically activates new neurons and propagates the accumulated training information to them. The neurons not only learn from the training data but also from other neurons. Different from a fixed large number of neurons, the valid neurons are increased gradually and the model performance is promoted accordingly. A prototype of active propagation RNN (APRNN) is implemented on Spark and multiple evaluations are carried out on language modeling tasks. The results show that compared with the fixed-neurons training, APRNN can reduce the training time while keeping a comparable accuracy. The improvement is more significant on large scale training corpora.