Unsupervised Learning by Examples: On-line versus Off-line
C. Van den Broeck, Peter Reimann · Physical Review Letters · 1996
We study both on-line and off-line unsupervised learning from p random patterns which are uniformly distributed on the N-sphere with the exception of a single symmetry breaking orientation $\mathit{B}$, along which they may be arbitrarily distributed. Supervised learning from the same kind of patterns is included as a special case. In the thermodynamic limit $N\ensuremath{\rightarrow}\ensuremath{\infty}$ with $\ensuremath{\alpha}\phantom{\rule{0ex}{0ex}}=\phantom{\rule{0ex}{0ex}}p/N$ fixed we calculate the overlap $R\left(\ensuremath{\alpha}\right)\phantom{\rule{0ex}{0ex}}=\phantom{\rule{0ex}{0ex}}\mathit{B}\ifmmode \dot{}\else \.{}\fi{}\mathit{J}/|\mathit{J}||\mathit{B}|$ between the unknown ``true'' $\mathit{B}$ and the optimal ``Bayes'' hypothesis $\mathit{J}$ with particular emphasis on the small and large \ensuremath{\alpha} asymptotics and the phenomenon of retarded learning. Finally, we identify a cost function whose minimum reproduces the off-line Bayes overlap.