Dynamics of online gradient descent learning
A C C Coolen, Reimer Kühn, Peter Sollich · 2005
Abstract In this chapter we consider learning in systems with continuous real-valued outputs, rather than those with binary outputs as in the previous chapter. A natural way of constructing learning algorithms for such systems is by gradient descent on some appropriate error measure. This error measure should tell us by how much student and teacher output differ. A popular choice is the squared deviation between actual and desired outputs; carrying out gradient descent on this measure for multilayer networks yields the well-known error backpropagation algorithm. The results of our analysis are therefore of relevance to practical neural network learning.