Cost-sensitive classifier evaluation
Robert C. Holte, Chris Drummond · 2005
Evaluating classifier performance in a cost-sensitive setting is straightforward if the operating conditions (misclassification costs and class distributions) are fixed and known. When this is not the case, evaluation requires a method of visualizing classifier performance across the full range of possible operating conditions. This paper reviews the classic technique for classifier performance visualization -- the ROC curve -- and argues that it is inadequate for the needs of researchers and practitioners in several important respects. It then shows that a different way of visualizing classifier performance -- the cost curve introduced by Drummond and Holte at KDD'2000 -- overcomes these deficiencies. A software package supporting all the cost curve analysis described in this paper is available by contacting the first author.