Estimating Prediction Qualities without Ground Truth: A Revisit of the Reverse Testing Framework
Dheeraj Bhaskaruni, Fiona Patricia Moss, Chao Lan · 2018
To evaluate prediction qualities of machine learning models, it is typically assumed testing samples are labeled. However, testing labels are not always available in practice. A traditional solution is to approximate prediction qualities on testing samples by the qualities on labeled training samples. But this may be limited in that it completely ignores testing samples. In this paper, we present a new approach to estimate prediction qualities on unlabeled testing sample, based on the reverse testing framework [1]. We evaluate the approach with various quality metrics in classification and anomaly detection tasks, and over numerous real-world data sets. Experimental results show the proposed approach gives a more accurate estimate of prediction qualities on testing sample than those on training samples.