Recursive M-estimation, nonlinear regression and neural network learning with dependent observations
Chung‐Ming Kuan, Halbert White · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · 1990
1.allowing for moderate dependence in the underlying stochastic process {Z,}, relying on mixingale convergence results of McLeish (1975).These results extend recent results of Englund, Hoist and Ruppert (1988).We specialize further in order to establish the properties of three implementations of the RM procedure applicable to the nonlinear regression model ~the "simple," "quick" and "modified" RM procedures.This permits us to generalize certain results of Albert and Gardner (1967), Ljung (1977), Ruppert (1983), Ljung and Soderstrom (1983), Metivier and Priouret (1984) and White (1989).Because the (extended) Kalman filter for a particular system coincides with the modified RM procedure, our results rigorously establish the consistency and asymptotic normality of this extended Kalman filter in a setting somewhat more general than previously available.Finally, the nonlinear regression results are applied to the estimation of parameters in a leading neural network model, considerably generalizing previously available results for network learning (e.g., White, 1989).The paper is organized as follows.In Section 2 we provide conditions ensuring the strong consistency and asymptotic normality of the general RM m-estimation algorithm.In Section 3 we introduce implementations of the RM algorithm suitable for use in the nonlinear regression problem and provide conditions establishing the consistency and asymptotic normality of these methods.Section 4 contains the neural network application, and Section 5 contains a summary and a discussion of directions for further research.A mathematical appendix contains the proofs of all results. ASYMPTOTIC PROPERTIES OF THE RM M-ESTIMATORThe ordinary differential equation (ODE) method for establishing consistency of recursive estimators introduced by Ljung (1977) and followed here makes use of certain interpolated processes.Given a sequence {a t e IR + ), let z, =£'ln tfi,T = 0. Define the piecewise linear