Accelerating facial motion capture with video-driven animation transfer
José Serra, Mark Williams, Lucio Moser · 2022
We describe a hybrid pipeline that leverages: 1) video-driven animation transfer [Moser et al. 2021] for regressing high-quality animation under partially-controlled conditions from a single input image, and 2) a marker-based tracking approach [Moser et al. 2017] that, while more complex and slower, is capable of handling the most challenging scenarios seen in the capture set. By applying the most suited approach to each shot, we have an overall pipeline that, without loss of quality, is faster and has less user intervention. We also improve the prior work [Moser et al. 2021] with augmentations during training to make it more robust for the Head Mounted Camera (HMC) scenario. The new pipeline is currently being integrated into our offline and real-time workflows.