Hold-out Risk Bounds for Classifier Performance Evaluation
Mohak Shah, Sara Shanian · 2009
We present an empirical study of the gener-alization error bounds on the empirical risk of classifier on a test set. We show how this approach, by modeling the empirical risk as a binomial, can be used to obtain realistic con-fidence intervals that lie strictly in the [0, 1] interval. This is in contrast to the traditional confidence interval approach that impose an asymptotic Gaussian assumption on the em-pirical risk which rarely holds for low risk-values resulting in unrealistic estimates on the limits of the intervals. 1.