Projection method applications to problems in neural networks and signal reconstruction
Henry Stark, Shu-Jen Yeh · 1991
In this thesis we consider two fundamental problems in neural networks: (1) the dynamics of Hopfield net: and (2) efficient learning in feedforward net. Both problems submit to solutions by way of projection methods. We formulate the dynamics of the bi-level Hopfield net in terms of generalized projection algorithms, and its continuous-level counterpart in terms of convex projections. From this analysis, the stable states of the Hopfield net can be categorized into three classes (two for continuous-level nets). Each class of the stable states has a different relation to the patterns intended to be stored in the net. We also compare the memory storage capacity of the original outer-product storage rule and the projection storage rule experimentally. The protection rule has a larger capacity than the conventional outer-product rule. The Hopfield net is applied to image restoration. We develop an iterative learning-restoration neural net process for constrained optimization. The Hopfield net computes the parameter adjustment to meet the constraint in the learning phase, and the same net using a different set of biases computes the restored image in the restoration phase. We demonstrate the feasibility of this algorithm by computer simulation. The problem of efficient learning in multilayer feedforward nets is addressed using the projection-method principle. We develop a learning rule that proves to be much faster than the well-known backpropagation learning rule, especially in the case when the units operate in the strongly nonlinear range. By monitoring the learning history of projection method learning and backpropagation learning, we find that the projection method learning does not have the long stagnation period typical of backpropagation, and is less sensitive to system parameters. Finally the projection methods are used to solve a problem in image reconstruction that is important in many fields of application. The problem is to reconstruct a bandlimited signal (image) from its nonuniformly spaced samples. We develop an iterative algorithm and a one-step method that are both applicable to one or two dimensional signal reconstruction. The algorithms reduce to the well known Whittaker-Shannon-Kotelnikov sampling formula when the samples are uniformly spaced. (Abstract shortened with permission of author.)