A differential theory of learning for efficient statistical pattern recognition
II John Benjamin Hampshire · 1993
There is more to learning stochastic concepts for robust statistical pattern recognition than the learning itself: computational resources must be allocated and information must be obtained. Therein lies the key to a learning strategy that is efficient, requiring the fewest resources and the least information necessary to produce classifiers that generalize well. Probabilistic learning strategies currently used with connectionist (as well as most traditional) classifiers are inefficient, requiring high classifier complexity and large training sample sizes to ensure good generalization. An asymptotically efficient differential learning strategy is set forth, which guarantees Bayesian (i.e., minimum probability-of-error) discrimination with the minimum-complexity classifier. Moreover, differential learning guarantees the best generalization allowed by the choice of classifier paradigm as long as the training sample size is large. When the training sample size is small, differential learning usually guarantees the best generalization allowed by the choice of classifier paradigm. The theory is demonstrated in several real-world machine learning/pattern recognition tasks associated with optical character recognition, medical diagnosis, airborne remote sensing imagery interpretation, and adaptive digital telecommunications. These applications focus on the implementation of differential learning and illustrate its advantages and limitations in a series of experiments that complement the theory. The experiments demonstrate that differentially-generated classifiers consistently generalize better than their probabilistically-generated counterparts across a wide range of real-world learning-and-classification tasks. The discrimination improvements range from moderate to significant, depending on the statistical nature of the learning task and its relationship to the functional basis of the classifier used.