Learning algorithms and the shape of the learning surface in recurrent neural networks
Tatsumi Watanabe, Y. Uchikawa, Kazutoshi Gouhara · Systems and Computers in Japan · 1992
Abstract In recent years there has been a renewal of interest in recurrent neural networks (RNN) because learning algorithms for RNN have been derived by several independent groups. The network can express spatiotemporal patterns as the states of the neurons change with time. Since the network has connections with feedback loops between each neuron, it includes the majority of conventional neural network models. The characteristics of three supervised learning algorithms are discussed from the viewpoint of: (1) formulations of gradient of the total squared error function; and (2) the number of calculations and amount of storage space, respectively. Through computer simulations the differences of convergence for each learning algorithm are investigated. For the backpropagation through time algorithm it is shown that the learning surface of RNN has two specific shapes, i.e., hills and valleys, and the learning descends gently on the steepest gradient forward along the bottom of a curved valley. These characteristics are basically consistent with those of the multilayer neural networks (MNN) analyzed by Gouhara and others.