An Exactly Solvable Model of Unsupervised Learning
Michael L. Biehl · Europhysics Letters (EPL) · 1994
A model for unsupervised learning from N -dimensional data is studied. Random training examples are drawn such that the distribution of their overlaps with a vector B ∊ R N is a mixture of two Gaussians of unit width and a separation ρ. A student vector is generated by an on-line algorithm, using each example only once. The evolution of its overlap with B can be calculated exactly in the thermodynamic limit N → ∞ . As a specific example, a learning algorithm closely related to Oja's rule is investigated. Its dynamics and approach to the stationary solution are solved for both a constant and an optimally chosen time-dependent learning rate. For the latter, the limits of small and infinitely large separation ρ of the peaks are considered. In both limits the analysis suggests the use of an asymptotic (1/ p )-decay for the learning rate, where p is the number of training examples. In the large-separation limit, the typical number of examples needed for successful learning is found to be ( p/N ) ∝ ρ -2 , which coincides with a recent result for supervised learning from Gaussian mixtures.