A New Coarse-To-Fine 3D Face Reconstruction Method Based On 3DMM Flame and Transformer: CoFiT-3D FaRe
Bao Quoc Phan Nguyen, Hung Thanh Nguyen, Ngoc Quoc Ly · 2023
The process of 3D face reconstruction has the remarkable capability to recover 3D information from a single facial image, enabling its utilization in accelerated 3D modeling, facial animation, and enhanced face recognition performance. The prevailing technique employs deep learning Convolutional Neural Networks (CNNs) to forecast intermediary 3D facial structures, such as 3D Morphable Model (3DMM) parameters or UV maps. While these methods demonstrate promise, they still encounter limitations with occlusions and depend heavily on pre-existing linear 3DMM, which restricts the utilization of input image data. In this paper, we introduce CoFiT-3D FaRe, a novel Coarse-to-Fine methodology for reconstructing 3D human faces. This approach makes use of the Dual Vision Transformer network (DaViT) and 3DMM Flame. The DaViT proficiently captures both local features and global context within the images, effectively overcoming limitations imposed by CNNs. In the initial coarse stage, the DaViT is employed to regress 3DMM parameters from the input image, facilitating the reconstruction of 3D human face’s shape and texture at a coarse level. While the 3DMM provides valuable three-dimensional information, its linear nature imposes constraints on achieving a realistic representation. To tackle this issue, the subsequent fine reconstruction stages extract more intricate information from the image and refine the outcomes of the initial coarse reconstruction. The paper employs detail reconstruction block of DECA (Detailed Expression Capture and Animation) for precise shape reconstruction and introduces an innovative approach for accurate texture reconstruction. This enables the restoration of occluded regions and the retention of intricate facial features. Our experiments on the NoW evaluation set demonstrate competitive outcomes in both shape and texture reconstruction when compared to state-of-the-art (SOTA) solutions.