Stereo Matching with Nonparametric Smoothness Priors in...
Transcript of Stereo Matching with Nonparametric Smoothness Priors in...
Stereo Matching with Nonparametric Smoothness Priors in Feature Space
Brandon M. Smith1, Li Zhang1, and Hailin Jin2
1 University of Wisconsin – Madison
2 Adobe Systems Incorporated1
Motivation
State-of-the-art two-view stereo methods9 out of top 10 employ image segmentation
UW-Madison Computer Graphics and Vision Group2
UW-Madison Computer Graphics and Vision Group
Motivation
Image segmentation problems
3
UW-Madison Computer Graphics and Vision Group
Motivation
Segmentation artifacts in video: temporal instability
1 of 5 input views 2nd-order smoothness method with segmentation[Woodford et al. CVPR ’08]
4
UW-Madison Computer Graphics and Vision Group
Inspiration: Adaptive Support Weight
Yoon & Kweon, Locally adapt. support-weight approach for vis. corr. search, CVPR ‘05
Intensity-encoded weightsClose-up views of matching window
5
UW-Madison Computer Graphics and Vision Group
Inspiration: Adaptive Support Weight
Can we incorporate this idea into a global inference algorithm?
Large, weighted smoothness nbrhoodClose-up views of matching window
6
UW-Madison Computer Graphics and Vision Group
Inspiration: Sparse Neighborhoods
O. Veksler, Stereo Correspondence by Dynamic Programming on a Tree, CVPR ‘05
Image close-up views Common smoothness neighborhood structure
Sparse tree smoothness neighborhood structure
Minimum spanning tree
algorithm
7
UW-Madison Computer Graphics and Vision Group
Our Approach
Minimum spanning tree
algorithm
Most important edgesLarge, weighted smoothness nbrhood
Global inference using large, sparse smoothness neighborhoods
8
Problem Formulation
UW-Madison Computer Graphics and Vision Group
2sm1sm21ph21 ,, DDDDDD
by minimizing:
compute disparity maps, and1D 2D
Given a stereo image pair, and , 1I 2I
9
Energy Minimization Function
2sm1sm21ph21 ,, DDDDDD
UW-Madison Computer Graphics and Vision Group
Photo consistency term[Kolmogorov & Zabih ECCV ‘02]
21ph , DD
Smoothness (regularization) terms
2sm1sm DD
10
UW-Madison Computer Graphics and Vision Group
Spatial Smoothness Terms
21ph21 ,, DDDD 2sm1sm DD
• 2nd-order smoothness priors
Previous global methods:• 1st-order smoothness priors
Another approach:• Kernel density estimation
Large neighborhood 11
Large neighborhood UW-Madison Computer Graphics and Vision Group
Kernel Density Estimation
dg
dd
d
qp
Nq p
p,qw
g(x)
x
p||
1 qqgcx
cc
c
xx
xNpp g
qp
Weight based on proximity, color between p,q [Yoon & Kweon CVPR`05]
21ph21 ,, DDDD 2sm1sm DD
12
Kernel function for disparity
Large neighborhood UW-Madison Computer Graphics and Vision Group
Upperbound Approximation
Nq p
p,qw
qp
21ph21 ,, DDDD 2sm1sm DD
dg
dd
d
qp
p||
1 qqgcx
cc
c
xx
xNp
g(x)
x
p g
Weight based on proximity, color between p,q [Yoon & Kweon CVPR`05]
Ip
Dsm
13
Kernel function for disparity
min( λ |dp dq|, τ)
Truncated linear difference function
UW-Madison Computer Graphics and Vision Group
Optimization Challenge
21ph21 ,, DDDD 2sm1sm DDsm is dense, expensive to minimize
p,qw
Nq pIp
Dsm
min( λ |dp dq|, τ)
Solution: use a sparse approximation
Large neighborhood 14
UW-Madison Computer Graphics and Vision Group
Sparse Graph Approximation
p,qw
Nq pIp
Dsm
min( λ |dp dq|, τ)
sm is dense, expensive to minimize
Solution: use a sparse graph approximation
DenseGraph
SparseGraph
15
UW-Madison Computer Graphics and Vision Group
Sparse Graph Approximation
DenseGraph
SparseGraph
1st Spanning Tree
ResidualGraph
2st Spanning Tree Nth Spanning Tree…
16
UW-Madison Computer Graphics and Vision Group
Dense smoothness neighborhood
Minimum spanning tree
algorithm
Sparse smoothness neighborhood
Graph Edges On Real Images
17
UW-Madison Computer Graphics and Vision Group
0% pixels connected 82% pixels connected 95% pixels connected 98% pixels connected 100% pixels connected
Connection to Image Segmentation
Kruskal’s minimum spanning tree algorithm
C. Zahn. Graph-theoretic methods for detecting and describinggestalt clusters. IEEE Trans. on Computing, 1971.
18
Results on Stereo Images
UW-Madison Computer Graphics and Vision Group
Left input image
Ground truth
Multi-view graph cuts result[Kolmogorov & Zabih, ‘02]
4.82% bad pixels
Our result
3.41% bad pixels 25.06% bad pixels
Second-order smoothness[Woodford et al. ’08]
19
UW-Madison Computer Graphics and Vision Group
2.48% bad pixels 4.21% bad pixels
Klaus et al. ‘06 results Multi-view graph cuts[Kolmogrov & Zabih ‘02]
(tailored parameters)
Left input image Ground truth depth
Results on Stereo Images
20
3.29% bad pixels
Our results(tailored parameters)
Results on Videos
UW-Madison Computer Graphics and Vision Group21
Future Work
UW-Madison Computer Graphics and Vision Group22
• Automatic parameter estimation (scale in kernel function)
• generalizes to any feature vector (not just x,y,r,g,b) explore other feature vectors
p,qw
• Better handle View-dependent brightness inconsistencies