Auto-Encoder Pre-Training of Segmented-Memory Recurrent Neural Networks
Stefan Glüge, Ronald Böck, Andreas Wendemuth · The European Symposium on Artificial Neural Networks · 2013
The extended Backpropagation Through Time (eBPTT) learning algorithm for Segmented-Memory Recurrent Neural Networks (SMRNNs) yet lacks the ability to reliably learn long-term dependencies. The alternative learning algorithm, extended Real-Time Recurrent Learn- ing (eRTRL), does not suffer this problem but is computational very in- tensive, such that it is impractical for the training of large networks. The positive results reported with the pre-training of deep neural networks give rise to the hope that SMRNNs could also benefit of a pre-training proce- dure. In this paper we introduce a layer-local pre-training procedure for SMRNNs. Using the information latching problem as benchmark task, the comparison of random initialised and pre-trained networks shows the beneficial effect of the unsupervised pre-training. It significantly improves the learning of long-term dependencies in the supervised eBPTT training.