Differential-algebraic equations and singular perturbation methods in recurrent neural learning
Ricardo Riaza, Pedro J. Zufiria · Dynamical Systems · 2003
This paper introduces a mathematical framework based on dynamical system theory, differential-algebraic equations (DAEs) and singularly perturbed (SP) systems, oriented to the analysis and design of on-line schemes for fixed point recurrent neural learning. New schemes proposed in this framework make it possible to relax some common assumptions in usual recurrent backpropagation (RBP) implementations. The scope of the work is not necessarily restricted to gradient-based adaptation methods, the results being applicable to more general learning strategies. The presented models clarify the relative timescaling between the network dynamics and the adaptation process in on-line techniques, including adjoint-based approaches. Certain restrictions on the learning speed are formalized through a 'rate of learning' limit appearing in the DAE/SP setting. Local convergence is rigorously stated, and certain Newton-based stabilization techniques are proposed regarding global issues in the presence of bifurcations, which typically introduce severe difficulties in common RBP methods. Some simulation examples concerning the synthesis of associative memories illustrate the applicability of the proposed techniques.