Riemannian structure of some new gradient descent learning algorithms
R.E. Mahoney, Robert C. Williamson · 2002
We consider some generalizations of the classical LMS learning algorithm including the exponentiated gradient (EG) algorithm. We show how one can develop these algorithms in terms of a prior distribution over the weight space. Our framework subsumes the notion of "link-functions". Differential geometric methods are used to develop the algorithms as gradient descent with respect to the natural gradient in the Riemannian structure induced by the prior distribution. This provides a Bayesian Riemannian interpretation of the EG and related algorithms. We relate our work to that of Amari (1985, 1997, 1998) and others who used similar tools in a different manner. Simulation experiments illustrating the behaviour of the new algorithms are presented.