Photo-reflective sensing for parametric 3D facial expression reconstruction in head-mounted display
Yuki Nakabayashi, Fumihiko Nakamura, Katsutoshi Masai, Maki Sugimoto · Computers & Graphics · 2026
Reconstructing the 3D facial expressions of head-mounted display (HMD) wearers is essential for natural avatar communication in virtual reality (VR). Camera-based methods achieve high fidelity but require heavy processing and raise privacy concerns. In contrast, non-imaging sensors are lightweight and privacy-preserving, but they provide only sparse features. We propose a reconstruction system that learns high-dimensional 3D facial representations from camera images during training and performs inference using only compact photo-reflective sensors embedded in the HMD. This design combines the expressiveness of camera-based supervision with the efficiency and privacy of sensor-based operation at inference time. Experimental results show that the system reconstructs 3D facial expressions from sensor data alone at inference time. Accuracy improves more from diverse HMD attachment conditions than from additional samples collected under a single attachment condition. A dedicated jaw-pose predictor and lightweight personalization with small wearer-specific datasets further reduce reconstruction error. A sensor ablation experiment further shows that sensor placement within a region is more important than regional coverage, and that glabella and nose sensors are particularly critical for reconstruction. Furthermore, we show that wear-invariant pretraining improves cross-wear robustness under one-set calibration. In this cohort, a calibration analysis identifies informative expression classes and yields a reduced-expression calibration protocol, in which users perform the first K expressions in a cohort-derived common order; with K = 20 , this protocol matches the full 30-expression calibration baseline.