sketches 0105: Markerless Facial Motion Capture using Texture Extraction and Nonlinear Optimization
Eugene Vendrovsky, Ivan Neulander, Hues Studios · 2005
The use of nonlinear optimization for estimating facial animation parameters is quite popular, as evidenced by [Paterson and Fitzgibbon 2003] and [Williams 2005]. However, these approaches include precise camera calibration and model tracking either as prerequisites or as part of the problem space while ours requires only approximations. We use a polygonal facial model driven by a control rig consisting of 10-50 scalar parameters that deform the model by blending between rest poses or driving a muscle simulation. Any scalar whose variation continuously affects the appearance of any part of the face is a suitable rig parameter. Our solver adjusts the rig parameters in an attempt to minimize a discrepancy metric between successive frames. We calculate this metric by constructing textures that capture the camera projection of the reference footage onto the deformed model at each frame, taking visibility into account. The textures are efficiently computed using a specialized tool called Primitex, which is integrated into our solver.