A proposal for comparative evaluation of automatic annotation for geo-referenced documents
Paul David Clough, Mark Sanderson · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · 2004
INTRODUCTION Organised evaluation campaigns such as the Text Retrieval Conference (TREC) for information retrieval [9], the Document Understanding Conference (DUC) for document summarization [7], and the Message Understanding Conference (MUC) for information extraction [3] have not only proven to be an important and effective stimulus for research, but also served to bring together members of the academic and industrial research communities. These campaigns have resulted in large-scale evaluations in which different approaches and techniques can be compared through the provision of common resources and evaluation strategies. A core task for Geographical Information Systems (GIS) is identifying and disambiguating geographical references (see, e.g. the SPIRIT and Geo-X-Walk projects). Previous workshops such as the HLT/NAACL 2003 Workshop on Analysis of Geographic References demonstrated a wide variety of different solutions to this problem, but methods were evaluated on diffe