Generalization in Fully Connected Committee Machines
Holm Schwarze, John Hertz · Europhysics Letters (EPL) · 1993
We study supervised learning in a fully connected committee machine trained to implement a rule of the same structure. The generalization error as a function of the number of training examples per weight is calculated within the annealed approximation. For binary weights we find a discontinuous transition from poor to perfect generalization. Beyond this transition metastable states exist even for large training sets. The scaling of the order parameters with the number of hidden units depends on the size of the training set. For continuous weights we find a discontinuous transition from a committee-symmetric solution to one with specialized hidden units.