On the weight dynamics of recurrent learning.
Schiller, Ulf D., Jochen J. Steil · UCL Discovery (University College London) · 2003
We derive continuous-time batch and online versions of the recently introduced efficient O(N 2 ) training algorithm of Atiya and Parlos (2000) for fully recurrent networks. A mathematical analysis of the respective weight dy- namics yields that efficient learning is achieved although relative rates of weight change remain constant due to the way errors are backpropagated. The result is a highly structured network where an unspecific internal dynamical reservoir can be distinguished from the output layer, which learns faster and changes at much higher rates. We discuss this result with respect to the recently introduced echo state and liquid state networks, which have similar structure.