Extension of Backpropagation through Time for Segmented-memory Recurrent Neural Networks
Stefan Glüge, Ronald Böck, Andreas Wendemuth · 2012
We introduce an extended Backpropagation Through Time (eBPTT) learning algorithm for SegmentedMemory Recurrent Neural Networks. The algorithm was compared to an extension of the Real-Time Recurrent Learning algorithm (eRTRL) for these kind of networks. Using the information latching problem as benchmark task, the algorithms’ ability to cope with the learning of long-term dependencies was tested. eRTRL was generally better able to cope with the latching of information over longer periods of time. On the other hand, eBPTT guaranteed a better generalisation when training was successful. Further, due to its computational complexity, eRTRL becomes impractical with increasing network size, making eBPTT the only viable choice in these cases.