On separable nonlinear least squares algorithms for neuro-fuzzy modular network learning
Eiji Mizutani, James Weldon Demmel · 2003
This paper focuses on separable nonlinear least squares algorithms arising in a class of so-called modular networks, describing how special architectural features can be exploited to develop efficient algorithms for a neuro-fuzzy model that has a modular architecture comprising multiple "local-expert" multilayer perceptrons (MLPs). In particular, we show that our structure-exploiting block-arrow least squares algorithm, that takes advantage of full Gauss-Newton model Hessian for all expert MLPs, can converge faster in both time and epoch than block-diagonal approximate Hessian-based algorithms when the nonlinear model has multiple outputs. In simulation, we demonstrate several variants of a dogleg trust-region algorithm with different model Hessians, showing trade-offs between the amount of Hessian information and convergence speed using a real-world nonlinear regression application.