Benefits of gain: speeded learning and minimal hidden layers in back-propagation networks
John K. Kruschke, Javier R. Movellan · IEEE Transactions on Systems Man and Cybernetics · 1991
The gain of a mode in a connectionist network is a multiplicative constant that amplifies or attenuates the net input to the node. The benefits of adaptive gains in back-propagation networks are explored. It is shown that gradient descent with respect to gain greatly increases learning speed by amplifying those directions in weight space that are successfully chosen by gradient descent on weights. Adaptive gains also allow normalization of weight vectors without loss of computational capacity, and the authors suggest a simple modification of the learning rule that automatically achieves weight normalization. A method for creating small hidden layers by making hidden node gains compete according to similarities between nodes in an effect to improve generalization performance is described. Simulations show that this competition method is more effective than the special case of gain decay.>