Adaptive algorithms for accelerated PCA from an augmented Lagrangian cost function
Chanchal Chatterjee, Zhengjiu Kang, Vwani Roychowdhury · 2003
We derive and discuss new adaptive algorithms for principal component analysis (PCA) that are shown to converge faster than the traditional PCA algorithms due to Oja, Sanger and Xu. Since online applications demand faster convergence and an automatic selection of gains, we employ nonlinear optimization techniques to present new algorithms to solve these problems. We first present an unconstrained objective function, which can be minimized to obtain the principal components. We derive adaptive algorithms from this objective function by using: (1) gradient descent, (2) steepest descent, (3) conjugate direction, and (4) Newton-Raphson methods. We also provide a discussion on the landscape of the objective function, and present a convergence proof of the adaptive gradient descent PCA algorithm using stochastic approximation theory. Extensive experiments with stationary and nonstationary multidimensional Gaussian sequences show the faster convergence of the new algorithms over the traditional gradient descent methods.