Validating Label Consistency in NER Data Annotation
Qingkai Zeng, Mengxia Yu, Wenhao Yu, Tianwen Jiang, Meng Jiang · 2021
Data annotation plays a crucial role in ensuring your named entity recognition (NER) projects are trained with the correct information to learn from.Producing the most accurate labels is a challenge due to the complexity involved with annotation.Label inconsistency between multiple subsets of data annotation (e.g., training set and test set, or multiple training subsets) is an indicator of label mistakes.In this work, we present an empirical method to explore the relationship between label (in-)consistency and NER model performance.It can be used to validate the label consistency (or catch the inconsistency) in multiple sets of NER data annotation.In experiments, our method identified the label inconsistency of test data in SCIERC and CoNLL03 datasets (with 26.7% and 5.4% label mistakes).It validated the consistency in the corrected version of both datasets.