Components of variance in ROC analysis of CADx classifier performance
Robert F. Wagner, Heang‐Ping Chan, Joseph T. Mossoba, Berkman Sahiner, Nicholas Petrick · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · 1998
We analyze the contributions to the population variance of the area under the ROC curve in assessment of CADx classifier performance and consider a number of models for this variance. The models all contain a pure term or terms in the number of training samples, a pure term in the number of test samples, plus a term or terms representing their interaction. The subset of terms containing the number of test samples also provide a model for what we call the mean Wilcoxon variance based on a single data set. By this variance we mean a nonparametric estimate of the uncertainty in the ROC area obtainable from a single experiment. The remaining terms--i.e., the pure terms in the number of training samples--are not directly estimable without drawing additional training samples. We investigate whether they may be inferred indirectly using a resampling strategy. The current study is presented within the context of our previous work on finite-sample effects on classifier performance, and is related to recent work of others on Analysis of Variance in ROC analysis.