An Online Growing-and-Pruning Algorithm of a Feedforward Neural Network for Nonlinear Systems Modeling
Xin Guo, Weisheng Wang, Jie Zhang, Lishuang Gong · IEEE Transactions on Automation Science and Engineering · 2024
In recent decades, many researchers and practitioners have always been focusing on the optimal design approach of feedforward neural network (FNN). However, it is still a challenge that FNN has a compact structure, while meeting some special requirements. This paper investigates the interrelation of nodes and the diversity of samples in sliding windows based on previous work for nonlinear system modeling and develops a new adaptive growing-and-pruning algorithm of FNN (AGPA-FNN). More specifically, the basis of AGPA-FNN is to dynamically grasp the interactions between hidden nodes through the local sensitivity analysis and mutual information method, which can seamlessly prune the redundancy hidden nodes through the punishing mechanism for weight decay. Thus, AGPA-FNN can prevent breaking the network structure and forget the acquired knowledge due to the sudden change in weights. In learning algorithm, an effective learning technique is developed to accelerate the learning efficiency of the gradient descent method (AOGD) through keeping the diversity of samples and a suitable learning rate in each window. Experimental investigations show that the proposed AGPA-FNN can effectively self-adjust hidden nodes based on data changes and finally achieve a compact structure, while it can outperform other approaches in terms of generalization performance. Note to Practitioners—Nonlinear system modeling has always been a key and difficult problem in real-world. In recent decades, Neural network has become an indispensable technique of nonlinear system modeling due its universal approximation property and strong modeling ability. However, finding the trade-off between the network structure and the generalization capability to meet some prespecified requirement for neural network has always been the fundamental challenge. The existing methods are always neglecting the interrelation between the pruned neurons and other neurons in pruning process, and usually compensate for the weight of other neurons after pruning redundancy neurons, which may seriously disrupt the stability of feedforward neural network (FNN). Therefore, this paper focuses on simultaneously self-organizing and self-updating both the structure and parameters of FNN for data streams, and develops a new adaptive growing-and-pruning algorithm of FNN (AGPA-FNN) in a bid to make a trade-off between the network structure and property. AGPA-FNN can dynamically self-organize its structure and seamlessly prune the redundancy hidden nodes through the punishing mechanism for weight decay based on the interactions between nodes using the local sensitivity analysis and mutual information method. Furthermore, the online gradient descent algorithm is improved to enhance the learning efficiency of FNN by keeping the diversity of samples and the suitable learning rate in each window. Finally, the proposed AGPA-FNN is validated by several benchmark and real-word problems.