Facial Landmark Detection and Head Pose Estimation of Occluded Faces Based on Stable Diffusion Image Completion
Wenzhangzhi Guo, Long Vân Tran Ha, Joel C. Davies, Eitan Grinspun, Lüder A. Kahrs · IEEE Access · 2025
Face occlusion often makes it challenging to accurately detect facial landmarks and head pose. In this work, we present a pipeline to estimate facial landmarks and head pose when only a partial face is visible. For each input image, we first apply the Stable Diffusion Inpainting algorithm multiple times to generate 25, 50, or 100 inpainted images. We then apply face trackers and calculate the mean shape of extracted landmarks in each image. Lastly, we estimate the head pose with the iterative closest point method. For facial landmarks, we performed experiments on 406 images from publicly available datasets. For the four investigated face trackers, our proposed pipeline was able to reduce the landmark detection error by at least 2-3 times, with a reduction in average detection error from 14.0 pixels to 3.9 pixels. For head pose estimation, we performed experiments on different views of a 3D head model with head pose ranges from -60° to 60° and 2D face images. Across more than 3,300 occluded test images, the proposed pipeline obtained a head pose within 10° of the ground truth in most cases. In conclusion, we present a system that enhances the accuracy of facial landmark detection and head pose estimation under large occlusions, without training any new machine learning models.