The role of evaluation in the development of content-based retrieval techniques
Arjen P. de Vries · University of Twente Research Information · 2000
There are no satisfactory methods for measuring the effectiveness of multimedia search techniques. Precision and recall types of metrics have been used in some of the literature but are impractical due to the tedious process of measuring relevances. The process is even more complicated because of human subjectivity in tasks involving multimedia. Also, multimedia collections quickly grow very large, making evaluation expensive with respect to the required hardware. There seem to be no standard corpora or benchmark procedures. In this document, we review the evaluation approaches that have been taken in the literature about multimedia retrieval systems. Our review covers far from all related publications, but we believe the sample of papers studied is sufficiently large to draw our (sad) conclusions.