Face-swapping Based Data Augmentation for ID Document and Selfie Face Verification
Elmokhtar Mohamed Moussa, Ioannis Sarridis, Emmanouil Krasanakis, Nathan Ramoly, Symeon Papadopoulos, Ahmad Montaser Awal, Lara Younes · 2025
In this work, we focus on identity verification by matching selfies to identity (ID) documents depicting faces. When applied to this task, state-of-the-art deep face verification models tend to underperform due to lack of documents spe-cific visual features in the public datasets, and suffer from skin tone biases due to representation imbalances of skin tone groups in training data. To address both issues, we introduce a framework that generates ID documents con-taining synthetic face images to augment training datasets. Specifically, we transform selfies into document-style images by combining style banks of document templates and face-swapping generative models. Experiments on a public and a proprietary real-world dataset reveal that fine-tuning state-of-the-art face verification models with the proposed methodology yields 7.96% improvement in verification ac-curacy, while requiring only 25% of the original training data. Furthermore, improvements occur across all skin tone groups, including darker skin tones; though it is notori-ously hard to perform accurate verification for this group, we achieve more than 50% relative reduction in false acceptance and false rejection rate gaps w.r.t. lighter skin tones.