Statistical mechanics of on-line learning and generalization
Michael L. Biehl, Nestor Caticha · 2002
Introduction In trying to understand how artificial neural networks (ANN) learn from examples, a variety of questions can be addressed which may require very different approaches. When asking about the typical properties of large ANN, the framework of Statistical Mechanics (SM) provides the natural set of tools as it was developed in order to obtain macroscopic properties emerging from microscopic interactions among a large number of units. The aim of this survey is to introduce the reader to the SM theory of online training of feed--forward neural networks and their generalization ability. The main characteristic of on--line learning is that training examples are dealt with one at a time, as opposed to off--line or memory based methods, where learning is guided by the minimization of a cost function which incorporates possibly all of the data. From a statistical physics point of view, the distinction is between systems which can be thought of as being in a state of thermal eq