Training recurrent neural networks with leap-frog
Johann Erich Wolfgang Holm, N.J.H. Kotze · 2002
Nonlinear recurrent neural networks are used as theoretical models to model and predict behavior of nonlinear systems such as electrical and mechanical loads and systems. Recurrent networks implement dynamic system models more efficiently than their feedforward counterparts. The efficiency of recurrent nets stems from the infinite number of state and state space trajectories that are exploited to enhance the storage capacity with respect to the weight space dimensionality. Training of recurrent networks is achieved by applying a classical optimization algorithm for training recurrent neural networks with hidden recurrent states. The leap-frog algorithm does not make use of excessive iterations or tedious line search algorithms to obtain the optimum weight vector of the recurrent network and exhibits the ability to escape for, shallow local minima. Leap-frog is used in off-line mode, when the identification model or predictor is trained, before the neural network is applied in adaptive mode. A comparison is drawn between leap-frog, online gradient descent, and block-mode gradient descent by using a theoretical nonlinear model with known (stable) states.