Motion Retargeting from Human in Video to 3D Characters with Different Skeleton Topology
Hao Guo, Shujie Li, Wei Jia · 2024
Motion retargeting from videos to 3D virtual character is a challenging task in computer vision and computer graphics. A solution is first to extract the 3D motion sequences from videos using human pose estimation algorithm and then retarget to 3D virtual character by skeleton motion retargeting algorithm. However, manual skeleton mapping is often necessary because of the difference in skeleton topology between pose estimation and 3D virtual character, introducing uncertainty and potentially compromising the semantics of the original motion. Additionally, the results of pose estimation contain noise while existing skeleton motion retargeting algorithms do not consider the influence of noise inputs in their design. In response to the above issues, we propose a novel motion retargeting framework and introduce a new motion retargeting network named CTBRnet (Combined Transformer Bi-LSTM Retargeting network). The framework leverages existing algorithm to generate paired training data, providing supervision to CTBRnet and enabling CTBRnet to learn potential skeleton mapping. The framework utilizes human motion videos as input and generates retargeting results for virtual characters, which are driven by joint rotations aligned with the virtual character's skeletal topology. Our proposed network combines transformer and Bi-LSTM, effectively capturing stable high-dimensional motion features from noisy results of pose estimation in both global and local perspectives. The experimental results show that our framework can achieve motion retargeting from videos to 3D virtual character with different skeleton topology, and compared with existing skeleton motion retargeting networks, CTBRnet reduces joint position errors while enhancing the smoothness.