Variable Projection Method and Levenberg-Marquardt Algorithm for Neural Network Training
Cheol-Taek Kim, Ju-Jang Lee, Hyejin Kim · Proceedings of the Annual Conference of the IEEE Industrial Electronics Society · 2006
An optimal solution for single hidden layered feedforward neural network (SLFN) is proposed. SLFN can be considered as separable nonlinear least squares problem and variable projection (VP) method gives the approximation of the Jacobian matrix of the problem. The Jacobian calculation of VP-SLFN is suggested with simplified form. Based on this simplification, Levenberg-Marquardt (LM) algorithm for VP-SLFN is suggested and has faster convergence rate than LM algorithm without VP. Two numerical examples show the superiority than extreme learning machine and LM without variable projection