Spherical light field representation and reconstruction from omnidirectional imagery
Kai Gu · HAL (Le Centre pour la Communication Scientifique Directe) · 2025
This thesis reviews advanced omnidirectional imaging (ODI) systems, light field capture and rendering pipelines, and the latest scene reconstruction paradigm, Radiance Field (RF). It addresses key challenges encountered in using RF for spherical light field representation and scene reconstruction. The primary contributions are in three aspects. First, we extend Neural Radiance Fields (NeRF) to support ODI inputs by incorporating an optimizable fisheye camera model. This allows for 360-degree scene reconstruction with ultra-wide-angle images, emphasizing the importance of spherical ray-sampling in such contexts. Second, to address the challenge of reconstructing 360-degree scenes from sparse ODI inputs, we combine vanishing point estimation with frequency encoding in an efficient hash-encoding framework, which significantly improves reconstruction quality without compromising efficiency. Finally, we enhance sparse-input scene reconstruction by integrating high-quality 2D image segmentation techniques. Using object mask matching from multiple views, we constrain objects within limited 3D spaces, improving scene reconstruction accuracy and achieving consistent 3D segmentation. These methods advance the RF-based reconstruction of 360-degree scenes, offering robust solutions for sparse input scenarios and enhancing visual experience in immersive image modalities.