Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy
Maharshi Gor, Kellie Webster, Jordan Lee Boyd-Graber · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
The goal of question answering (QA) is to answer any question.However, major QA datasets have skewed distributions over gender, profession, and nationality.Despite that skew, model accuracy analysis reveals little evidence that accuracy is lower for people based on gender or nationality; instead, there is more variation on professions (question topic).But QA's lack of representation could itself hide evidence of bias, necessitating QA datasets that better represent global diversity.1 For NQ, we only consider questions with short answers. 2 https://cloud.google.com/natural-language/docs/ analyzing-entities 3 We analyze the dev fold, which is consistent with the training fold (Table 1 and 2), as we examine accuracy.