Natural gradient learning algorithms for nonlinear systems
Zhao Junsheng, Xia Jianwei, Zhuang Guangming, Huasheng Zhang · 2015
Nonlinear systems are widely used models for function approximation in the regression problem. Like the neural networks, there exists many strange behaviors in the learning process of some nonlinear systems, such as the slow learning speed, the existence of the plateaus and so on. As is known, the natural gradient learning method can overcome these disadvantages effectively. In this paper, we first introduce a common nonlinear system and calculate the explicit expression of the Fisher information matrix. And then we introduce the natural gradient learning to the nonlinear system. Since it is difficult to calculate the inverse of the Fisher matrix when parameters are in the singular region, then we introduce the adaptive method to implement the natural gradient learning algorithms. At last, we give the explicit forms of the adaptive natural gradient learning algorithms and compare it with the conventional gradient descent method. Simulations show that the proposed adaptive natural gradient method which can avoid the plateaus effectively, have a good performance when RBF networks are used for nonlinear functions approximation.