Learning by examples from a nonuniform distribution
Peter Reimann, C. Van den Broeck · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · 1996
We present a general replica calculation for learning from examples generated by a nonuniform pattern distribution with a single symmetry-breaking orientation. Our results cover the three main learning scenarios: storage of patterns with random classifications by a perceptron, supervised learning from a teacher, and unsupervised learning. We show that for a perceptron the critical storage capacity ${\ensuremath{\alpha}}_{c}=2$ is completely independent of the pattern distribution provided it is point symmetric or provided the classification as \ifmmode\pm\else\textpm\fi{} 1 is unbiased. In a particular model for supervised learning we find that an ideal (Bayes) student learns most from a few examples if they are easy and from a large number if they are difficult. Learning based on the minimization of a specific class of (quadratic) cost functions is solved completely for all three scenarios.