TD-NeRF: Transfer Learning and Diffusion Regulation-Based NeRF for Scene Perception
Wentao Hu, Chenyu Tian, Long Wen, Huafeng Ding · IEEE Transactions on Instrumentation and Measurement · 2024
Scene perception based on 3-D reconstruction has been widely applied in various downstream applications for intelligent devices. The use of deep learning methods for scene perception has become increasingly prevalent. However, accurately perceiving environmental information with few-shot and purely image-based inputs presents a substantial challenge. In the absence of sufficient input guidance, the rendered outputs are prone to artifacts and overfitting, which can lead to suboptimal performance in the model’s scene perception capabilities. To address the issues, this study introduces novel transfer learning and diffusion regulation-based neural radiance fields (TD-NeRFs). First, this method employs transfer learning to further process a limited set of images and their corresponding sparse depth information, obtaining complete depth and standard deviation (std) images for network supervision. Second, a diffusion sampling module (DSM) standardizes the sampling process, guiding the accurate distribution of depth information. Third, a diffusion frequencies module (DFM) is utilized to suppress high-frequency signals during the initial stages of training, ensuring accurate low-frequency learning and preventing artifacts. Experiments on the official dataset show that TD-NeRF achieves peak signal-to-noise ratio (PSNR) values of 21.51 and 20.92 on the ScanNet and LLFF datasets, respectively. Moreover, on the ScanNet dataset, the depth-related root mean square error (RMSE) reaches 0.171. These results demonstrate that TD-NeRF exceeds the performance of various existing few-shot NeRF approaches.