Convergence for Receiver Operating Characteristic Curves and the Performance of Neural Networks
Stephen G. Alsing, Kenneth W. Bauer, Mark E. Oxley · International Journal of Smart Engineering System Design · 2002
Receiver operating characteristic (ROC) curves are commonly used for summarizing the performance of imperfect diagnostic systems. In the pattern recognition community, a commonly held assumption is that for the case of unlimited data, a ROC curve exists. In reality only finite data is available so only an approximate ROC curve can be constructed. As more (finite) data is added the approximate ROC curve is updated. Performing this process repeatedly yields a sequence of approximate ROC curves. We propose a family of metrics for comparing two ROC curves that enable a proof of convergence for this sequence of curves. This ROC convergence theorem is an important contribution because it provides a framework for the comparison of ROC curves and hence, the comparison of classifiers. We apply our proposed metrics to three real-world applications - the University of Wisconsin Breast Cancer Diagonsis problem, an automatic target recognition (ATR) problem, and an aircraft pilot workload problem. We demonstrate how our proposed metrics can eliminate the ambiguity that can result when the ROC curves of competing classifiers overlap. We close the paper with a discussion of the interpretation of the new metric.