Image retrieval: Large‐scale evaluation of cross‐language image retrieval systems
Paul David Clough · Bulletin of the American Society for Information Science and Technology · 2007
valuating the performance of information retrieval (IR) sys-E tems is an important part of the development process.For example, it is necessary to establish to what extent the system being developed meets the needs of end users, to show the effects of changing the underlying system or its functionality on system performance and to enable quantitative comparison between different systems and approaches.However, although most agree that evaluation is important in IR, much debate exists on exactly how this evaluation should be carried out.In this article we describe an initiative known as ImageCLEF (Cross-Language Evaluation Forum), a large-scale evaluation campaign with the aims of providing resources to the research community and of creating an environment in which to carry out and discuss the evaluation of image retrieval systems. EvaluationEvaluation of retrieval systems tends to focus on either the system or the user.Saracevic (1995) distinguishes six levels of evaluation for information systems that include IR systems: engineering level, w input level, rn processing level, output level, w use and user level and social level.For many years IR evaluation has tended to focus on the first three levels, predominately through the use of stanbenchmarks (or tesdreference collections) in a laboratory-style setting.rn