Revisiting Natural Gradient for Deep Networks
Razvan Pascanu, Yoshua Bengio · 2014
We evaluate natural gradient, an algorithm originally proposed in Amari (1997), for learning deep models. The contributions of this paper are as follows. We show the connection between natural gradient and three other recently proposed meth-ods: Hessian-Free (Martens, 2010), Krylov Subspace Descent (Vinyals and Povey, 2012) and TONGA (Le Roux et al., 2008). We empirically evaluate the robust-ness of natural gradient to the ordering of the training set compared to stochastic gradient descent and show how unlabeled data can be used to improve generaliza-tion error. Another contribution is to extend natural gradient to incorporate second error information alongside the manifold information. Lastly we benchmark this new algorithm as well as natural gradient, where both are implemented using a truncated Newton approach for inverting the metric matrix instead of using a di-agonal approximation of it. 1