Facilitating Reconciliation of Inter-Annotator Disagreements
Johann Stan, Dina Demner‐Fushman, Kin Wah Fung, Olivier Bodenreider, L. Owen Hill · AMIA · 2014
Development and evaluation of Natural Language Processing methods often requires text annotation. To gauge the difficulty of the task and increase the reliability and quality of annotations, researchers often recruit at least two annotators. The discrepancies in annotations by multiple annotators need to be identified and reconciled. We present a tool that identifies and helps reconciling and validating annotations in a widely used annotation tool Brat.