Function representation and approximation by neural networks.
David Shi Chen · Deep Blue (University of Michigan) · 1991
In this dissertation, we have investigated the representational power of multilayer feedforward neural networks both analytically and constructively. Our study reveals that the function representation by multilayer networks is flexible and general since it includes polynomial representation, spline representation, kernel based representation, projection based representation, and Fourier analysis representation. In essence, the representation by a multilayer network is a linear combination of nonlinear basis functions. These basis functions exhibit complex and flexible shape descriptive power. The precise shape, orientation, and location of each basis function are functions of the network weights; therefore, they are adaptively estimated via network learning algorithms. We have provided a new proof for the Universal Approximation Theorem for multilayer feedforward neural networks, based upon tools in the theory of measure and probability. The Theorem states that any multivariate (real valued) Borel function can be approximated to any desired accuracy by a network with three hidden layers. We have developed a robust back propagation algorithm that is able to train a network in the presence of erroneous data. The novelty of the new algorithm is the use of a new objective function whose shape varies with learning time. The knowledge of the noise model revealing the data quality is encoded into the shape of the time-varying objective function. The effect of each training data on the overall learning is weighted according to its quality. It is this novel strategy that allows the new algorithm to be stable under small noise perturbation and robust against gross errors in a training data. We have successfully applied the method of function approximation via multilayer feedforward neural networks to approximate a wide variety of univariate and bivariate functions, and to solve problems involving multivariate functions. Particularly, we have demonstrated that the new method is capable of performing discontinuity-preserving and outlier-removing surface reconstruction from noisy data--a challenging task in computer vision.