Rank quality for evaluating ccbr system performance
David B. Leake, Steven A. Bogaerts · 2007
Rank quality is a measure of the degree of match between a retrieved list of cases and the list that would be retrieved given perfect information. This dissertation advances that the rank quality measure distinguishes between the case retrieval performance of various case-based reasoning (CBR) systems more than the common combination of precision and efficiency. More effectively means that systems with different levels of performance can be correctly ordered by rank quality measurements, while precision and efficiency measurements suggest merely partial or even incorrect orderings. The concept of rank quality is intuitive, but a formal definition is deceptively difficult to design. Many formulations appear to have promise but are ultimately unsuitable for this task. A number of measurements related to the novel concept of distance granularity are useful in examining the problems of these formulations. The final rank quality formulation fully addresses these problems and is proven effective using both formal and experimental arguments. This dissertation begins with a brief introduction to CBR, followed by an introduction to rank quality. Rank quality is compared intuitively to precision and efficiency measures, and placed in context in CBR research. Various applications of rank quality are proposed. Following this, a number of possible formulations of rank quality are examined. This brings to the fore a number of interesting issues in case retrieval and evaluation, as well as some useful tools for this task. The final formulation of rank quality is presented and justified formally and experimentally. This formulation is compared experimentally with precision and efficiency, corresponding with the earlier intuitive discussion. Finally, rank quality is applied to the evaluation of missing attribute strategies.