Semantic-SRF: Sparse Multi-view Indoor Semantic Segmentation with Stereo Neural Radiance Fields
Cem Eteke, Jinpeng Zhang, Eckehard G. Steinbach · 2022
We study the effect of prior training of neural implicit networks on the task of semantic segmentation of room-scale scenes. For this, we train Stereo Radiance Field (SRF) on color and density information obtained from dense views of training scenes. In return, SRF learns general scene radiance, geometry and appearance features. We further extend SRF to Semantic-SRF that enables the decoder of SRF to predict semantics. Later we use sparse semantic views to fine-tune the implicit representations on novel-scenes. This allows us to extract scene-level semantics using only a few semantically labelled images. We show the efficacy of this approach on a photorealistic indoor room dataset, namely Replica. We compare the approach with another implicit network that does not use prior training, namely Semantic-NeRF. We demonstrate that as the number of sparse views decreases, the scene-level features enable the network to persist its performance.