A statistical mechanical analysis of a Bayesian inference scheme for an unrealizable rule
Glenn Marion, David Saad · Journal of Physics A Mathematical and General · 1995
Within a Bayesian framework we consider a system that learns from examples. In particular, using a statistical mechanical formalism, we calculate the evidence and two performance measures, namely the generalization error and the consistency measure, for a linear perceptron trained and tested on a set of examples generated by a nonlinear teacher. The teacher is said to be unrealizable because the student can never model it without error. In fact, our model allows us to interpolate between the known linear case and an unrealizable, nonlinear, case. A comparison of the hyperparameters which maximize the evidence with those that optimize the performance measures reveals that, when the student and teacher are fundamentally mismatched, the evidence procedure is a misleading guide to optimizing the performance measures considered.