Intrinsic Plasticity via Natural Gradient Descent
Klaus Neumann, Jochen J. Steil · 2012
Abstract. This paper introduces the natural gradient for intrinsic plas-ticity, which tunes a neuron’s activation function such that its output dis-tribution becomes exponentially distributed. The information-geometric properties of the intrinsic plasticity potential are analyzed and the im-proved learning dynamics when using the natural gradient are evaluated for a variety of input distributions. The applied measure for evaluation is the relative geodesic length of the respective path in parameter space. 1