A Multitask Network for Multiview Stereo Reconstruction: When Semantic Consistency-Based Clustering Meets Depth Estimation Optimization
Xin Huang, Shulei Zhang, Jiayi Li, Leiguang Wang · IEEE Transactions on Geoscience and Remote Sensing · 2024
We propose a novel network for Multi-View Stereo reconstruction in the field of remote sensing, which considers Clustering-based Semantic Consistency into depth estimation optimization, referred to as CSC-MVS. In this approach, high-level semantic information acquired from multiple views is utilized to construct semantic consistency and assist in guiding the optimization of the MVS network. Specifically, the Non-negative Matrix Factorization (NMF) branch and the Deep Spectral Decomposition (DSD) branch, are designed to generate local and global semantic guidance, respectively. We then propose an uncertainty multi-task optimization method to adaptively combine matching and semantic metrics. The performance of CSC-MVS is evaluated on representative benchmarks, including the WHU TLC dataset and LuoJia-MVS dataset, demonstrating its effectiveness and generality across diverse remote sensing scenarios. Comprehensive experimental results show that our CSC-MVS significantly improves the performance of various MVS baseline networks and achieves notable accuracy in depth reconstruction. We also conduct ablation studies to validate the rationality of each component, and sensitivity analysis to confirm the robustness and adaptability of our proposed method. The code is available at https://github.com/zsl-whu/csc-mvs.