Optimal initialization for multi-layer perceptrons
Y. Chen, Farokh Bastani · 2002
An optimal weight initialization algorithm for three-layer networks, called the Walsh-Hermite algorithm, is introduced. The weights between the input layer and the hidden layer are vectors of the Hadamard matrix. Each neuron of the hidden layer has a different discriminator function, which is a base function of the Hermite spline of degree three. The weights between the hidden layer and the third layer are scaled coefficients of Hermite interpolations. Experiments show that the convergence speed of the network with weights so initialized is quite close to that of the best initialized net. Compared to the optimal estimate training algorithm, the Walsh-Hermite algorithm is theoretically more reasonable, is cost effective, and does not confine the number of units in the hidden layer.>