Learning from minimum entropy queries in a large committee machine
Peter Sollich · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · 1996
In supervised learning, the rebundancy contained in random examples can be avoided by learning from queries. Using statistical mechanics, we study learning from minimum entropy queries in a large tree-committee machine. The generalization error decreases exponentially with the number of training examples, providing a significant improvement over the algebraic decay for random examples. The connection between entropy and generalization error in multilayer networks is discussed, and a computationally cheap algorithm for constructing queries is suggested and analyzed.