The TREC video retrieval evaluation (TRECVID): a case study and status report
Alan F. Smeaton, Wessel Kraaij, Paul Over · 2004
The TREC Video Retrieval Evaluation (TRECVID) is an annual international effort, funded by the US Advanced Research and Development Activity (ARDA) and the National Institute of Standards and Technology (NIST) to promote progress in content-based retrieval from digital video via open, metrics-based evaluation. Now beginning its fourth year, TRECVID aims over time to developboth a better understanding of how systems can effectively accomplish video retrieval and how one can reliably benchmark their performance. This paper is a case study in the development of video retrieval systems and their evaluation as well as a report on the TRECVID status to-date. After an introduction to the evolution of TRECVID over the past 3 years, we report on the most recent evaluation TRECVID 2003 in terms of the 4 tasks (shot boundary determination, high-level feature extraction, story segmentation and classification, search), the data (133 hours of US television news), the measures, the results obtained, and the approaches taken by some of the 24 participating groups. 1 Origins and evolution The impetus for the TREC video retrieval evaluation effort came from the observation that while quantities of digital video were growing at a rapidly increasing rate and interesting research was being done, there was no widely available basis for scientific comparison of approaches taken to content-based retrieval, namely no common training and test data, benchmark tasks, established evaluation procedures or measures. The TREC video retrieval evaluation was established in 2001 to explore the feasibility of addressing this need.