University at Buffalo at SemEval-2023 Task 11: MASDA–Modelling Annotator Sensibilities through DisAggregation
Michael W. Sullivan, Mohammed Yasin, Cassandra L. Jacobs · 2023
Modeling the most likely label when an annotation task is perspective-dependent discards relevant sources of variation that come from the annotators themselves.We present three approaches to modeling the controversiality of a particular text.First, we explicitly represented annotators using annotator embeddings to predict the training signals of each annotator's selections in addition to a majority class label.This method leads to reduction in error relative to models without these features, allowing the overall result to influence the weights of each annotator on the final prediction.In a second set of experiments, annotators were not modeled individually but instead annotator judgments were combined in a pairwise fashion that allowed us to implicitly combine annotators.Overall, we found that aggregating and explicitly comparing annotators' responses to a static document representation produced highquality predictions in all datasets, though some systems struggle to account for large or variable numbers of annotators.