Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation
Tomasso Caselli, Oana Inel · University of Groningen research database (University of Groningen / Centre for Information Technology) · 2018
This paper describes a crowdsourcing experiment on the annotation of plot-like structures in En- glish news articles. The CrowdTruth methodology and metrics have been applied to select valid annotations from the crowd. We further run an in-depth analysis of the annotated data by compar- ing it with available expert data. Our results show a valuable use of crowdsourcing annotations for such complex semantic tasks, and promote a new annotation approach that combines crowd and experts.