Convergence Analysis of Regularized Elman Neural Networks Under Relaxed Conditions
Qinwei Fan, Zhou Li, Jacek M. Zurada, Jigen Peng, Haiyang Li, Jian Wang, Dakun Yang · IEEE Transactions on Systems Man and Cybernetics Systems · 2024
Recurrent neural networks (RNNs) have been found to be a promising field of research for time series prediction, continuous-time system modeling, and discrete-time sequence data processing. The Elman network can be viewed as an RNN with a single layer of feedback connections and local storage units. However, the Elman network has two limitations that restrict its application. One is the overfitting phenomenon, which limits its generalization ability. The other is the complexity of the network structure, which makes its theoretical results incomplete and limits its practicality. The main contribution of this article is to propose a new Elman network model with a regularization term. First, we demonstrate the convergence of the backpropagation algorithm with weight decay regularization under some reasonable conditions. Second, contrary to the usual requirement for bounded weights, we show that such boundedness is no longer a necessary condition for convergence analysis. Third, we show that the requirements for learning rate and the stable set of error functions can also be relaxed. Finally, we provide simulation results to demonstrate the excellent performance of our proposed algorithm.