Finite-sample effects and resampling plans: applications to linear classifiers in computer-aided diagnosis
Robert F. Wagner, Heang‐Ping Chan, Berkman Sahiner, Nicholas Petrick, Joseph T. Mossoba · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 1997
This work provides an application and extension of the analysis of the effect of finite-sample training and test sets on the bias and variance of the classical discriminants as given by Fukunaga. The extension includes new results for the area under the ROC curve, Az. An upper bound on Az is provided by the so-called resubstitution method in which the classifier is trained and tested on the same patients; a lower bound is provided by the hold-out method in which the patient pool is partitioned into trainers and testers. Both methods exhibit a bias in Az with a linear dependence on the inverse of the number of patients Nt used to train the classifier; this leads to the possibility of obtaining an unbiased estimate of the infinite-population performance by a simple regression procedure. We examine the uncertainties in the resulting estimates. Whereas the bias of classifier performance is determined by the finite size of the training sample, the variance is dominated by the finite size of the test sample. This variance is approximately given by the simple result for an equivalent binomial process. A number of applications to the linear classifier are presented in this paper. More general applications, including the quadratic classifier and some elementary neural-network classifiers, are presented in a companion paper.