Disparity estimation from multi-view images and video: graph models and algorithms
C.‐C. Jay Kuo, Jong Dae Oh · 2008
In this work, we consider the problem of estimating the depth information from the following three scenarios: a stereo image pair, multi-view images, and stereo image sequences. For stereo image matching, we first propose a new disparity map estimation algorithm for slant and curved surfaces. In this situation, we focus on two properties of the disparity map. The first one is continuous disparity change inside an object while the second one is sharp disparity change between object boundaries. To exploit these two properties at the same time, two techniques are proposed to improve the performance of existing stereo matching algorithms. To address disparity discontinuity in object boundaries, we present a disparity estimation procedure, which consists of two steps: a greedy disparity filling algorithm and a least-squared-errors (LSE) fitting method. Furthermore, it is observed that the existing fronto-parallel model with color segmentation is built upon the piecewise constant surface approximation. This is however not efficient in approximating slanted or curved objects. We propose to use a piecewise linear surface model to represent 3-dimensional (3D) geometric structure for better surface modeling. The proposed stereo matching system with these two new components is evaluated with Middlebury data sets with excellent quantitative and qualitative results. Then, a new graph model for disparity estimation of multi-view images is investigated. Two performance metrics for algorithmic evaluation are considered. They are quality and complexity. To reduce complexity, a graph model whose computational time is independent of the number of input images is proposed. With this model, new smoothness and consistency terms in the energy function are added to maintain the quality of a disparity map. Experimental results show that the proposed algorithm offers good enough quality at a much lower complexity than existing methods. To further improve the quality of the disparity map in stereo matching, we propose an advanced framework based on new graph models. Along this direction, we divide stereo matching problem into 3 sub-problems: (1) disparity estimation for non-occlusion regions and occlusion detection, (2) disparity estimation for occlusion regions, and (3) post-processing of the disparity map. A three-step procedure is proposed to solve them sequentially. At the first step, we perform an initial matching and develop a new graph model using the ordering constraint to improve disparity values in non-occlusion regions and detect occlusion regions. At the second step, we determine disparity values in occlusion regions based on global optimization. Since the conventional segmentation-based stereo matching is not efficient in highly slanted or curved objects, we propose a post-processing technique for disparity map enhancement based on a 3-dimensional (3-D) geometric structure. The proposed three-step stereo matching procedure yields excellent quantitative and qualitative results with Middlebury data sets. Finally, disparity estimation and virtual view synthesis from stereo video inputs are examined. To enhance the overall performance, a two-stage algorithm for accurate and fast disparity estimation and occlusion handling is first presented. Then, a new virtual view synthesis method with a preprocessing algorithm is described. The preprocessing algorithm can remove false matched regions for disparity refinement effectively. This synthesis method can reduce the blurring and ghostly effects and provides an excellent tradeoff in terms of computational time and synthesized video quality.