Techniques for evaluating classifiers in application
Amy L. Magnus, Mark E. Oxley · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 2004
In gauging the generalization capability of a classifier, a good evaluation technique should adhere to certain principles. For instance, the technique should evaluate a selected classifier, not simply an architecture. Secondly, a solution should be assessable at the classifier’s design and, further, throughout its application. Additionally, the technique should be insensitive to data presentation and cover a significant portion of the classifier’s domain. Such principles call for methods beyond supervised learning and statistical training techniques such as cross validation. For this paper, we shall discuss the evaluation of a generalization in application. For illustration, we will present a method for the multilayer perceptron (MLP) that may be drawn from the unlabeled data collected in the operational use of a given classifier. These conclusions support self-supervised learning and computational methods that isolate unstable, nonrepresentational regions in the classifier.