Lecture8 - Stereo
-
Upload
jose-rafael -
Category
Documents
-
view
220 -
download
0
Transcript of Lecture8 - Stereo
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 1/67
6.801/866
Stereopsis
T. Darrell
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 2/67
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 3/67
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 4/67
text
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 5/67
Outline
• Reconstruction
• Rectification
• Early vs. Late
• Window Matching• Edge Matching
• Ordering Constraint
• More views
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 6/67
baseline
depth
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 7/67
baseline
depth
Triangulate on two images of the
same point to recover depth.
– Feature matching across views
– Calibrated cameras
Left Right
Last lecture: only needto match features across
epipolar lines…
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 8/67
Geometric Reconstruction
Geometric: midpoint of intersection of Op and O’p’
Multiple views?
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 9/67
Algebraic Reconstruction
Point must satisfy both projection equations:
and therefore:
solve using least squares.
No obvious geometric interpretation; well-defined for multiple views.
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 10/67
Nonlinear Reconstruction
Nonlinear technique:
search for point Q with minimal projection error:
(initialize with one of previous linear methods.)
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 11/67
Rectified image pair
• Choose normalized image
locations
• Common image plane
• Horizontal epipolar lines
• Correspondence search on
scanlines!
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 12/67
Pinhole Camera Model
Focal length f
Virtual
Image
Center of
projection
Image
plane
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 13/67
Pinhole Camera Model
),,( Z Y X P =
Virtual
Image
f −
O y
x
z
Image
plane
),( y x p =
1
1
1 Z
X
f x−=
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 14/67
Basic Stereo Derivations
1O y
x
z
f −
2O y
x
z
1 p
B
),,(1 Z Y X P =
2 p
B f x x Z ,,,of functionaasfor expressionDerive 21
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 15/67
Basic Stereo Derivations
),,(1 Z Y X P =1O y
x
z
f −
2O y
x
z
211
1
1
1
12
1
11 ,
x x
B f Z
Z B f x
Z B X f x
Z X f x
−
=⇒
−=+−=−=
B
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 16/67
Basic Stereo Derivations
y
x
z
Ray from
camera 2
1 x
Ray from
camera 1
2 x
d
B f Z
x xd
=
−=21 :disparitytheDefine
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 17/67
Stereo Vision
baseline
depth
),(),(
y xd
B f y x Z =
Z ( x, y) is depth at pixel ( x, y)
d ( x, y) is disparity
Left Right
Matching across scan lines
How do matching errors affect
the depth error?
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 18/67
Outline
Reconstruction
Rectification
• Early vs. Late
• Window Matching• Edge Matching
• Ordering Constraint
• More views
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 19/67
Consider human vision
Differences?
• active foveation
• image plane
• object recognition
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 20/67
Human vision is very accurate at relative depth
judgements.
Julez (1960): which happens first, recognition or
fusion?
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 21/67
Random dot stereograms
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 22/67
Random dot stereograms
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 23/67
Random dot stereograms
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 24/67
Correspondence is ambiguous
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 25/67
Marr-Poggio (1976)
Three constraints:
• compatibility• uniqueness
• continuity
Works well on RDS….but not so well on natural images…
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 26/67
Outline
Reconstruction
Rectification
Early vs. Late
• Window Matching• Edge Matching
• Ordering Constraint
• More views
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 27/67
Correspondence using window matching
Points are highly individually ambiguous…
More unique matches are possible with small regionsof image.
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 28/67
Correspondence using window matching
Left Right
error
disparity
scanline
Criterion function:
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 29/67
Sum of Squared (Pixel) Differences
Left Right
Lw Rw
Lw
L I
Rw
R I
mm
),( L L y x ),( L L yd x −
∑∈−−=
+≤≤−+≤≤−=
),(),(
2
2222
)],(),([),,(
:disparityof functionaasdifferenceintensitythemeasurescostSSDThe
},|,{),(
:functionwindowthedefineWe
pixels.of windows byingcorrespondareand
y xW vu R Lr
mmmmm
R L
m
vd u I vu I d y xC
yv y xu xvu y xW
mmww
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 30/67
Image Normalization
• Even when the cameras are identical models, there can bedifferences in gain and sensitivity.
• The cameras do not see exactly the same surfaces, so their
overall light levels can differ.• For these reasons and more, it is a good idea to normalize
the pixels in each window:
pixel Normalized ),(
),(ˆ
magnitudeWindow )],([
pixelAverage ),(
),(
),(),(
2
),(
),(),(),(
1
y xW
y xW vu y xW
y xW vu y xW
m
m
m
m
m
I I
I y x I y x I
vu I I
vu I I
−
−=
=
=
∑
∑
∈
∈
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 31/67
Images as Vectors
“Unwrap”
image to form
vector, usingraster scan order
Left Right
Lw Rw
Lw
row 1 m
m
Lwrow 3
m row 2 m
Each window is a vector
in an m2 dimensional
vector space.
Normalization makes
them unit length.
m
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 32/67
Image windows as vectors
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 33/67
Possible metrics
Lw
)(d w R Distance?
Angle?
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 34/67
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 35/67
Correspondence Using Correlation
Left Disparity Map
Images courtesy of Point Grey Research
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 36/67
Foreshortening
Window methods assume fronto-parallel surface at 3-D point.
Initial estimates of the disparity can be used to warp thecorrelation windows to compensate for unequal amounts of foreshortening in the two pictures [Kass, 1987; Devernay
and Faugeras, 1994].
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 37/67
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 38/67
Outline
Reconstruction
RectificationEarly vs. Late
Window Matching
• Edge Matching
• Ordering Constraint
• More views
bl i h i d h d
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 39/67
Problems with window methods
Patch too small?
Patch too large?
Can try variable patch size [Okutomi and Kanade],or arbitrary window shapes [Veksler and Zabih]
Should match between physically meaningfulquanties, and at multiple scales [Marr]…
M P i l i h
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 40/67
Marr-Poggio algorithm
Search for edges, a.k.a. “zero crossings”: (more
during edge detection lectures…)
O d i i
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 41/67
Ordering constraint
“It is reasonable to assume that the order of matching image
features along a pair of epipolar lines is the inverse of the
order of the corresponding surface attributes along the
curve where the epipolar plane intersects the observed
object’s boundary.”
This is the so-called ordering constraint introduced by [Baker
and Binford, 1981; Ohta and Kanade, 1985].
O d i t i t
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 42/67
Ordering constraint
O d i t i t
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 43/67
Ordering constraint
Oops!
DP b d h
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 44/67
DP-based search
Ordering constraint == smooth path through match graph.
Consider path’s cost over a graph whose nodes correspond to pairs of left and right image features, and arcs represent
matches between left and right intensity profile intervals
bounded by the features of the corresponding nodes
S h O C d
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 45/67
Search Over Correspondences
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Three cases: – Sequential – cost of match
– Occluded – cost of no match
– Disoccluded – cost of no match
Dynamic Programming
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 46/67
Dynamic Programming
Occluded Pixels
Left scanline
Di s
- o c c l u d e d P i x e l s
Ri gh t s c a nl i n
e
End
Start
Dynamic programming
yields the optimal
path through grid.This is the best set of
matches that satisfy
the orderingconstraint
Dynamic Programming
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 47/67
Dynamic Programming
1=i 1 1 112Π
22Π2= j2 2 22=i
32Π
3 3 33=i
1−t C t C 1+t C
Principle of Optimality for an n-stage assignment problem:
)(maxarg)( 1 iC jC t ijit −Π=
Dynamic Programming
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 48/67
Dynamic Programming
1=i 1 1 1
2)2( =t b2= j2 2 22=i
3 3 33=i
1−t C t C 1+t C
Principle of Optimality for an n-stage assignment problem:
)(maxarg)( 1 iC jC t ijit −Π=
Dynamic Programming
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 49/67
Dynamic Programming
1=i 1 1 1
2
3
2
13Π
23Π
33Π
22=i
3= j3 33=i
1−t C t C 1+t C
Principle of Optimality for an n-stage assignment problem:
)(maxarg)( 1 iC jC t ijit −Π=
Dynamic Programming
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 50/67
Dynamic Programming
1=i 1 1 1
2
3
2
1)3( =t b
22=i
3= j3 33=i
1−t C t C 1+t C
Principle of Optimality for an n-stage assignment problem:
)(maxarg)( 1 iC jC t ijit −Π=
Dynamic Programming
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 51/67
Dynamic Programming
1)1( =t b1= j1=i 1 1 1
2
3
2
1)3(=
t b
2)2( =t b22=i
3 33=i
1−t C t C 1+t C
Principle of Optimality for an n-stage assignment problem:
)(maxarg)( 1 iC jC t ijit −Π=
Dynamic Programming
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 52/67
Dynamic Programming
1=i 1 1 1
2
3
2
3
2
3
2−T C 1−T C T C
Back-chaining recovers the optimal path and its cost:
...),(),(),(maxarg **
1
***
T T T T T T T jT sb s sC C jC s ===−
2=i
3=i
)2(
2*
*
T T
T
C C
s
=
=
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 53/67
Stereo Matching with Dynamic
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 54/67
Programming
Occluded Pixels
Left scanline
Di s - o c c l u d e d P i x
e l s
Ri gh t s c a nl i n e
Scan across grid
computing optimal
cost for each nodegiven its upper-left
neighbors.
Backtrack from theterminal to get the
optimal path.Terminal
Stereo Matching with Dynamic
i
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 55/67
Programming
Occluded Pixels
Left scanline
Di s - o c c l u d e d P i x
e l s
Ri gh t s c a nl i n e
Scan across grid
computing optimal
cost for each nodegiven its upper-left
neighbors.
Backtrack from theterminal to get the
optimal path.Terminal
Stereo Matching with Dynamic
P i
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 56/67
Programming
Occluded Pixels
Left scanline
Di s - o c c l u d e d P i x
e l s
Ri gh t s c a nl i n e
Scan across grid
computing optimal
cost for each nodegiven its upper-left
neighbors.
Backtrack from theterminal to get the
optimal path.Terminal
Stereo Matching with Dynamic
P i
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 57/67
Programming
Occluded Pixels
Left scanline
Di s - o c c l u d e d P i x
e l s
Ri gh t s c a nl i n e
Scan across grid
computing optimal
cost for each nodegiven its upper-left
neighbors.
Backtrack from theterminal to get the
optimal path.Terminal
Stereo Matching with Dynamic
P i
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 58/67
Programming
Occluded Pixels
Left scanline
Di s - o c c l u d e d P i x
e l s
Ri gh t s c a nl i n e
Scan across grid
computing optimal
cost for each nodegiven its upper-left
neighbors.
Backtrack from theterminal to get the
optimal path.Terminal
Stereo Matching with Dynamic
P i
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 59/67
Programming
Occluded Pixels
Left scanline
Di s - o c c l u d e d P i x
e l s
Ri gh t s c a nl i n e
Scan across grid
computing optimal
cost for each nodegiven its upper-left
neighbors.
Backtrack from theterminal to get the
optimal path.Terminal
DP vs Edges
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 60/67
g
Edges:
DP:
• Which method is better? – Edges are more “meaningful” [Marr]…but hard to find!
– Edges tend to fail in dense texture (outdoors)
– Correlation tends to fail in smooth featureless areas
Computing Correspondences
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 61/67
p g p
Both methods fail for smooth surfaces
There is currently no good solution to thecorrespondence problem
Outline
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 62/67
Reconstruction
RectificationEarly vs. Late
Window Matching
Edge Matching
Ordering Constraint
• More views
Three (calibrated) views
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 63/67
Trinocular Stereo ResultsWhy this camera
arrangement?
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 64/67
Trinocular stereo system available from Point
Gray Research for $5K (circa ’97)
More views reduce ambiguity
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 65/67
Middlebury stereo page
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 66/67
http://www.middlebury.edu/stereo/
Outline
8/2/2019 Lecture8 - Stereo
http://slidepdf.com/reader/full/lecture8-stereo 67/67
Reconstruction
RectificationEarly vs. Late
Window Matching
Edge Matching
Ordering Constraint
More views
[ Most figures adapted from Forsythe and Ponce ]