Fixed Point Propagation: A New Way To Train Recurrent Neural Networks Using Auxiliary Variables
Somjit Nath · ERA: Education and Research Archive (University of Alberta) · 2019
Recurrent neural networks (RNNs), along with their many variants, provide a powerful tool for online prediction in partially observable problems. Two issues concerning RNNs, however, are the ability to capture long-term dependencies and long training times. There have been a variety of strategies to improve training in RNNs, particularly by approximating an algorithm called Real-Time Recurrent Learning. These strategies, however, can still be computationally expensive and focus computation on computing gradients back- in-time. In this work, we show that learning the hidden state in RNNs can be framed as a fixed-point problem. Using this formulation, we provide an asynchronous fixed-point iteration update that significantly improves run-times and stability of learning the state update.