Convergence of the Wake-Sleep Algorithm
Shiro Ikeda, Шун-ичи Амари, Hiroyuki Nakahara · 1998
The W-S (Wake-Sleep) algorithm is a simple learning rule for the models with hidden variables. It is shown that this algorithm can be applied to a factor analysis model which is a linear version of the Helmholtz machine. But even for a factor analysis model, the general convergence is not proved theoretically. In this article, we describe the geometrical understanding of the W-S algorithm in contrast with the EM (ExpectationMaximization) algorithm and the em algorithm. As the result, we prove the convergence of the W-S algorithm for the factor analysis model. We also show the condition for the convergence in general models. 1 INTRODUCTION The W-S algorithm[5] is a simple Hebbian learning algorithm. Neal and Dayan applied the W-S algorithm to a factor analysis model[7]. This model can be seen as a linear version of the Helmholtz machine[3]. As it is mentioned in[7], the convergence of the W-S algorithm has not been proved theoretically even for this simple model. From the similarity of...