Components of variance in ROC analysis of CADx classifier performance: II. Applications of the bootstrap
Robert F. Wagner, Heang‐Ping Chan, Berkman Sahiner, Nicholas Petrick, Joseph T. Mossoba · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 1999
We review components-of-variance models for the uncertainty in estimates of the area under the ROC curve, Az, for the case of classical discriminants where we wish the uncertainty to generalize to a population of training cases as well as to a population of testing cases. A key observation from our previous work facilitates the use of resampling strategies to analyze a finite data set and classifier in terms of the components-of-variance models. In particular, we demonstrate the use of the statistical bootstrap in combination with a four-term variance model to solve for the contributions of the uncertainty in Az that result from a given finite training sample, a given finite test sample, and their interaction. At the same time one obtains an expression from which one can predict the change in uncertainty in estimates of Az that would result from a given change in the number of training samples and change in the number of test samples. This expression provides a quantitative design tool for estimating the size that would be required in a larger pivotal study from the results of a smaller pilot study for the purpose of achieving a desired precision in Az and the desired generalizability.