Live4D: A Real-time Capture System for Streamable Volumetric Video
Yifeng Zhou, Shuheng Wang, Wenfa Li, Chao Zhang, Li Rao, Pu Cheng, Yi Xu, Jinle Ke, Wenduo Feng, Wen Zhou, Hao Xu, Yukang Gao, Yang Ding, Wenxuan Tang, Shaohui Jiao · 2023
Volumetric video holds promise for virtual and augmented reality (VR/AR) applications but faces challenges in interactive scenarios due to high hardware costs, complex processing and substantial data streams. In this paper, we introduce Live4D, a cost-effective, real-time volumetric video generation and streaming system using an RGB-only camera setup. We propose a novel deep implicit surface reconstruction algorithm, that combined neural signed distance field with observed truncated signed distance field to generate the watertight meshes with low latency. Moreover, we achieve a robust non-rigid tracking method that provides temporal stability to the meshes while resisting tracking failure cases. Experimental results show that Live4D achieves a performance of 24fps using mid-range graphic cards and exhibits an end-to-end latency of 95ms. The system enables live streaming of volumetric video within a 20Mbps bandwidth requirement, positioning Live4D as a promising solution for real-time 3D vision content creation in the growing VR/AR industry.