A lyapunov method for correlational learning in two layer neural networks
Todd W. Troyer, Morris W. Hirsch · 1993
A class of two layer networks is defined. There are feedforward connections between input and output layers and lateral connections within the output layer. A single input to the network consists of fixed pattern of activation in the input layer. The lateral weights are fixed and symmetric and constrained so that for any given input pattern x and feedforward weight matrix W, the activation dynamics within the output layer has a globally attracting equilibrium $ z = F(x, W).$ Input patterns are chosen ergodically from a fixed, finite set and each feedforward weight is changed or learned according to the average correlation of activity at either end of the connection. The main result of the dissertation is to produce a Lyapunov function for the averaged learning equations for this class of neural networks. In addition, two saturation results are proved concerning existence of solutions in which outputs are near their limiting values. These results are applied to networks with two different patterns of lateral connectivity. In the first application, uniform lateral inhibition is used to implement soft competition within a layer of category detecting nodes; algebraic conditions on the resulting categories are derived. In the second network, both the input layer and output layer consist of a one dimensional ring of nodes and the lateral connectivity is center-surround. A topographic solution for this network is a locally stable configuration of the feedforward weight matrix in which the input/output function F commutes with translation. A geometrical representation of existence conditions for such solutions is presented. Assuming a high degree of symmetry and synchrony, the method is extended to include the amplitude equations for networks with oscillatory dynamics. Finally, it is shown how the structure of the Lyapunov function suggests a general approach to a broader class of such two layer networks.