Deep Learning on Object-Centric 3D Neural Fields
Pierluigi Zama Ramirez, Luca De Luigi, Daniele Sirocchi, Adriano Cardace, Riccardo Spezialetti, Francesco Ballerini, Samuele Salti, Luigi Di Stefano · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024
In recent years, Neural Fields (NFs) have emerged as an effective tool for encoding diverse continuous signals such as images, videos, audio, and 3D shapes. When applied to 3D data,NFs offer a solution to the fragmentation and limitations associated with prevalent discrete representations. However, given thatNFs are essentially neural networks, it remains unclear whether and how they can be seamlessly integrated into deep learning pipelines for solving downstream tasks. This paper addresses this research problem and introducesnf2vec, a framework capable of generating a compact latent representation for an inputNFin a single inference pass. We demonstrate thatnf2veceffectively embeds 3D objects represented by the inputNFs and showcase how the resulting embeddings can be employed in deep learning pipelines to successfully address various tasks, all while processing exclusivelyNFs. We test this framework on severalNFs used to represent 3D surfaces, such as unsigned/signed distance and occupancy fields. Moreover, we demonstrate the effectiveness of our approach with more complexNFs that encompass both geometry and appearance of 3D objects such as neural radiance fields.