Towards Data Submissions for Shared Tasks: First Experiences for the Task of Text Alignment ?
Martin Potthast, Steve Goering, Paolo Rosso, Benno Stein · CLEF (Working Notes) · 2015
This paper reports on the organization of a new kind of shared task that outsources the creation of evaluation resources to its participants. We intro- duce the concept of data submissions for shared tasks, and we use our previous shared task on text alignment as a testbed. A total of eight evaluation datasets have been submitted by as many participating teams. To validate the submitted datasets, they have been manually peer-reviewed by the participants. Moreover, the submitted datasets have been fed to 31 text alignment approaches in order to learn about the datasets' difficulty. The text alignment implementations have been submitted to our shared task in previous years and since been kept operational on the evaluation-as-a-service platform TIRA.