Evaluation of targeted dataset collection on racial equity in face recognition
Rachel S. Hong, Tadayoshi Kohno, Jamie Morgenstern · 2023
Algorithmic audits of industry face recognition models have recently incentivized companies to diversify their data collection methods, which in turn has reduced error disparities along demographic lines, such as gender or race. We argue that it is important to understand exactly how various forms of targeted data collection mitigate performance disparities in these updated face recognition models. We propose an empirical framework to assess the impact of additional dataset collection targeted towards various racial groups. We apply our framework to three racially-annotated benchmark datasets using three standard face recognition models. Our findings empirically validate the notion that the introduction of data from the demographic group with the initially-lowest performance improves performance on that group significantly more than adding from other groups. We also observe that in all settings, the introduction of data from a previously omitted group does not harm the performance of other groups. Furthermore, investigation of feature embeddings reveals that performance increases are associated with a larger separation among images of different identities. Despite the commonalities we observe across datasets, we also find key differences: for example, in one dataset, training on one racial group generalizes well across all groups. These differences speak to the criticality of re-applying empirical evaluation methods, such as the methods in this work, when introducing new datasets or models.