Small nets and short paths : optimising neural computation
Marcus R. Frean · ERA · 1990
This thesis explores two aspects of optimisation in neural network research.The question of how to find the optimal feed-forward neural network architecture for learning a given binary classification is addressed.The so-called constructive approach is reviewed whereby intermediate, hidden, units are built as required for the particular problem.Current constructive algorithms are compared, and three new methods are introduced.One of these, the Upstart algorithm, is shown to outperform all other constructive algorithms of this type.This work led on to the ancillary problem of finding a satisfactory procedure for changing the weight values of an individual unit in a network.The new thermal perceptron rule is described and is shown to compare favorably with its competitors.Finally the spectrum of possible learning rules is surveyed.Neurobiologically inspired algorithms for mapping between spaces of different dimensions are applied to a classic optimisation problem, the Travelling Salesman Problem.Two new methods are described that can tackle the general symmetric form of the TSP, thus overcoming the restriction on other neural network algorithms to the geometric case.Part I Pattern Classification by Perceptrons Chapter 1 Constructive neural network algorithms 1.1 Overview Distinguishing one pattern from another is among the most fundamental of operations for any system which responds to an environment.Learning to do so is among the most important of abilities.Connectionism has developed novel ways of thinking about both these problems, and has enjoyed a certain amount of success at solving them.However there are problems with the way these connectionist networks learn: the following chapters are about ways these limitations might be overcome by novel learning strategies.