Fusing Structural and Appearance Features for 3D Layout Estimation
Weidong Zhang, Xueke Hu, Ying Liu, Yuquan Gan · 2023
3D layout estimation aims to infer the overall spatial structure of the indoor scene from an input image, and it reflects the distribution of the indoor dominant planes in a 3D space. Recent researches on layout estimation mainly learn the layout clues and features based on the visual appearance from the color images. However, little attention is paid to the depth maps that can provide the 3D structural information of the dominant planes and the relationships to the indoor objects, e.g., a bed is usually placed against a wall. Inspired by this, we first adopt a monocular depth estimation module and learn the structural feature from the estimated depth map. Then we fuse the structural feature and the appearance feature from the color image to learn the surface parameters for 3D layout inference. During training, we propose to add virtual planar obstacle in the estimated depth map, which can promote the network to put more attention on global information and also augment the training samples. Experimental results show that our method achieves state-of-the-art performance, and it proves that the depth map can provide favorable and reliable geometric clues for layout estimation.