Statistical Mechanics of the Mixture of Experts
Kukjin Kang, Jong–Hoon Oh · 1996
We study generalization capability of the mixture of experts learning from examples generated by another network with the same architecture. When the number of examples is smaller than a critical value, the network shows a symmetric phase where the experts does not specialize. Unon crossing the critical point, the system undergoes a continuous phase transition to a symmetry breaking phase where the gating network partitions the input space effectively and each expert is assigned to an appropriate subspace. We also find that the hierarchical mixture of experts shows multiple phase transitions. 1 Introduction Recently there has been considerable interest among neural network community in techniques that integrate the collective predictions of a set of networks[1, 2, 3, 4]. The mixture of experts [1, 2] is a well known example which implements the philosophy of divide-and-conquer. However, there have been little efforts to evaluate generalization capability of these modular approaches th...