Quantifying the expertise of classifiers using 4-value logic
Amy L. Magnus, Mark E. Oxley · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 2002
An intelligent agent---defined as an autonomous, adaptive, cooperative computer program---must credibly represent its expertise in negotiations with peer agents. Given an agent-based classifier, the determination of where in the domain the classifier is an expert must be explicitly stated. Likewise, where the classifier is confused should also be represented. Currently, an error measures provides an estimate of the relative size of the expertise and confusion sets, but error does not offer a distinct opinion on an untruthed feature vector's membership---i.e., whether its classification is based on specific information, conjecture or chance. We propose the theory for estimating the complete membership of a classifier's expertise sets and confusion sets. From these sets, we construct a 4-value classifier that hypothesizes for each new feature vector whether its classification can be made confidently or not. Examples are given that demonstrate the utility of this theory using multilayer perceptrons.