Expectation-conjugate gradient: An alternative to EM
Ruslan Salakhutdinov, Sam T. Roweis, Zoubin Ghahramani · Cambridge University Engineering Department Publications Database · 2002
We show a close relationship between bound optimization (BO) algorithms such as Expectation-Maximization and direct optimization (DO) algorithms such as gradient-based methods for parameter learning. We identify analytic conditions under which BO algorithms exhibit QuasiNewton convergence behavior, and conditions under which these algorithms possess poor, first-order convergence. In particular, for the EM algorithm we show that if a certain measure of the proportion of missing information is small, then EM exhibits Quasi-Newton behavior; when it is large, EM converges slowly. Based on this analysis, we present a new Expectation-Conjugate-Gradient (ECG) algorithm for maximum likelihood estimation, and report empirical results, showing that, as predicted by the theory, ECG outperforms EM in certain cases.