Synthetic data generator: understanding human face & body via image synthesis
Yu Yin · 2023
The synthesis of data has long been a valuable resource for training machine learning models, offering reliability and control while reducing the reliance on real-world data collection. This is particularly pertinent to the realms of human face and body synthesis, where concerns regarding model fairness and ethical deployment hold paramount importance. Nevertheless, the challenge persists in generating convincing, identity-preserving, and high-quality digitized humans within the domains of 2D and 3D image synthesis. This dissertation embarks on a comprehensive exploration of the potential to comprehend human behavior by recreating it, presenting its findings through three distinct and interconnected sections: 1) Enhancing 2D Image Generation: We delve into the realm of 2D image generation models and their symbiotic relationship with facial applications, such as landmark localization and face recognition tasks. A strong correlation is observed between super- resolution (SR) and the localization of minute facial features. We propose integrated frameworks that leverage mutual benefits between face alignment and SR, thereby amplifying the performance of both tasks. Furthermore, we highlight how face frontalization serves as an effective augmentation technique, enhancing face recognition accuracy under extreme pose scenarios. Additionally, we extend our exploration to encompass GAN inversion and editing techniques, facilitating the creation of more authentic and lifelike facial representations. 2) Advancing 3D Parametric Generation: Shifting focus to 3D parametric generation models, we explore their instrumental role in facilitating human body pose and shape estimation. In healthcare, technological advancements aimed at monitoring body and behavioral patterns during sleep and rest are imperative. Nonetheless, challenges emerge when accounting for occlusion caused by blankets during rest. Addressing this, we introduce an attention- based generation module to mitigate uncertainties arising from occluded body parts. This module generates uncovered modalities, iteratively refining the current estimations via a cyclic process. 3) Pioneering 3D Nerf-Based Generative Models: Our exploration further extends to the cutting- edge realm of 3D Nerf-based Generative models, focused on generating high-quality images with consistent 3D geometry fidelity. A novel and universally applicable method is presented, enabling precise fine-tuning of NeRF-GAN models. This groundbreaking approach yields remarkably detailed animations of real subjects from a single image, thus bridging the gap between data synthesis and high-fidelity animation. In essence, this dissertation contributes to the advancement of knowledge by unraveling the intricacies of human behavior through data synthesis. Through an integrated approach across 2D and 3D domains, it opens avenues for enhanced machine learning training, healthcare monitoring, and realistic animation, thereby bridging the gap between synthetic and real-world understanding.--Author's abstract