Multi-View Geometry: Find Corresponding Points...
Transcript of Multi-View Geometry: Find Corresponding Points...
![Page 1: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/1.jpg)
Multi-View Geometry:Find Corresponding Points
(New book: Ch7.4, 7.5, 7.6Old book: 11.3-11.5)
Credit for materials: Trevor Darrell, Berkeley, C280, Marc Pollefeys, UNC/ETH-Z, CS6320 S012, Andrew Zisserman, MVG Book
Guido GerigCS-GY 6643, Spring 2016
![Page 2: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/2.jpg)
Excellent Website: http://vision.middlebury.edu/stereo/
![Page 3: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/3.jpg)
Stereo reconstruction: main steps
– Calibrate cameras– Rectify images– Compute disparity– Estimate depth
Grauman
![Page 4: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/4.jpg)
Stereo reconstruction: main steps
– Calibrate cameras– Rectify images– Compute disparity– Estimate depth
Grauman
![Page 5: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/5.jpg)
Correspondence problem
Multiple match hypotheses satisfy epipolar constraint, but which is correct?
Figure from Gee & Cipolla 1999 Grauman
![Page 6: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/6.jpg)
Correspondence problem• Beyond the hard constraint of epipolar geometry,
there are “soft” constraints to help identify corresponding points– Similarity– Uniqueness– Ordering– Disparity gradient
Grauman
![Page 7: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/7.jpg)
Correspondence problem• Beyond the hard constraint of epipolar geometry,
there are “soft” constraints to help identify corresponding points– Similarity– Uniqueness– Ordering– Disparity gradient
• To find matches in the image pair, we will assume– Most scene points visible from both views– Image regions for the matches are similar in appearance
Grauman
![Page 8: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/8.jpg)
Your basic stereo algorithm
Adapted from Li Zhang Grauman
![Page 9: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/9.jpg)
Your basic stereo algorithm
For each epipolar line:
Adapted from Li Zhang Grauman
![Page 10: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/10.jpg)
Your basic stereo algorithm
For each epipolar line:For each pixel in the left image
Adapted from Li Zhang Grauman
![Page 11: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/11.jpg)
Your basic stereo algorithm
For each epipolar line:For each pixel in the left image
• compare with every pixel on same epipolar line in right image
Adapted from Li Zhang Grauman
![Page 12: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/12.jpg)
Your basic stereo algorithm
For each epipolar line:For each pixel in the left image
• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
Adapted from Li Zhang Grauman
![Page 13: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/13.jpg)
Your basic stereo algorithm
For each epipolar line:For each pixel in the left image
• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
Adapted from Li Zhang Grauman
![Page 14: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/14.jpg)
Your basic stereo algorithm
For each epipolar line:For each pixel in the left image
• compare with every pixel on same epipolar line in right image
• pick pixel with minimum match cost
Improvement: match windows• This should look familiar...• E.g. SSD, correlation etc.
Adapted from Li Zhang Grauman
![Page 15: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/15.jpg)
Stereo matching
• Search is limited to epipolar line (1D)• Look for “most similar pixel”
?
![Page 16: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/16.jpg)
Stereo matching algorithms• Match Pixels in Conjugate Epipolar Lines
– Assume brightness constancy– This is a tough problem– Numerous approaches
• dynamic programming [Baker 81,Ohta 85]• smoothness functionals• more images (trinocular, N-ocular) [Okutomi
93]• graph cuts [Boykov 00]
– A good survey and evaluation: – http://vision.middlebury.edu/stereo/
![Page 17: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/17.jpg)
Correspondence using Discrete Search
![Page 18: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/18.jpg)
Compare intensities pixel-by-pixel
Comparing image regions
I(x,y) I´(x,y)
Census
Similarity measures
125 126 125
127 128 130
129 132 135
0 0 0
0 1
1 1 1
(Real-time chip from TYZX based on Census)
only compare bit signature
![Page 19: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/19.jpg)
Sum of Squared Differences (SSD)
![Page 20: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/20.jpg)
Example
![Page 21: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/21.jpg)
Example ctd
![Page 22: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/22.jpg)
Example image pair – parallel cameras
![Page 23: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/23.jpg)
First image
![Page 24: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/24.jpg)
Second image
![Page 25: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/25.jpg)
Intensity profiles
• Clear correspondence between intensities, but also noise and ambiguity
![Page 26: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/26.jpg)
Dense correspondence algorithmParallel camera example – epipolar lines are corresponding rasters
![Page 27: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/27.jpg)
Dense correspondence algorithmParallel camera example – epipolar lines are corresponding rasters
![Page 28: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/28.jpg)
Dense correspondence algorithmParallel camera example – epipolar lines are corresponding rasters
epipolar line
![Page 29: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/29.jpg)
Dense correspondence algorithm
Search problem (geometric constraint): for each point in the left image, the corresponding point in the right image lies on the epipolar line (1D ambiguity)
Disambiguating assumption (photometric constraint): the intensity neighbourhood of corresponding points are similar across images
Measure similarity of neighbourhood intensity by cross-correlation
Parallel camera example – epipolar lines are corresponding rasters
epipolar line
![Page 30: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/30.jpg)
Correspondence problem
Source: Andrew Zisserman
Neighborhood of corresponding points are similar in intensity patterns.
![Page 31: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/31.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Slide the window along the epipolar line until w.w’ is maximized.
![Page 32: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/32.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Slide the window along the epipolar line until w.w’ is maximized.
![Page 33: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/33.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Slide the window along the epipolar line until w.w’ is maximized.
![Page 34: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/34.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Slide the window along the epipolar line until w.w’ is maximized.
![Page 35: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/35.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Slide the window along the epipolar line until w.w’ is maximized.
![Page 36: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/36.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Slide the window along the epipolar line until w.w’ is maximized.
![Page 37: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/37.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Slide the window along the epipolar line until w.w’ is maximized.
![Page 38: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/38.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Slide the window along the epipolar line until w.w’ is maximized.
![Page 39: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/39.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Slide the window along the epipolar line until w.w’ is maximized.
![Page 40: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/40.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Normalized Correlation: minimize θ instead.
Slide the window along the epipolar line until w.w’ is maximized.
![Page 41: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/41.jpg)
Correlation Methods (1970--) F&P book new: 7.4, old 11.3
Normalized Correlation: minimize θ instead.
Slide the window along the epipolar line until w.w’ is maximized.2Minimize |w-w’|.
![Page 42: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/42.jpg)
Cross-correlation of neighbourhood regions
• left and right windows encoded as vectors w and w’• zero-mean vectors (𝑤𝑤 − �𝑤𝑤) and (𝑤𝑤′ − �𝑤𝑤’) • Normalized cross-correlation:
• Advantage: Invariant to intensity differences: Invariant to affine intensity transformation 𝐼𝐼′ = 𝛼𝛼𝐼𝐼 + 𝜇𝜇
![Page 43: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/43.jpg)
Cross-correlation of neighbourhood regions
epipolar line
• left and right windows encoded as vectors w and w’• zero-mean vectors (𝑤𝑤 − �𝑤𝑤) and (𝑤𝑤′ − �𝑤𝑤’) • Normalized cross-correlation:
• Advantage: Invariant to intensity differences: Invariant to affine intensity transformation 𝐼𝐼′ = 𝛼𝛼𝐼𝐼 + 𝜇𝜇
![Page 44: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/44.jpg)
Cross-correlation of neighbourhood regions
epipolar line
• left and right windows encoded as vectors w and w’• zero-mean vectors (𝑤𝑤 − �𝑤𝑤) and (𝑤𝑤′ − �𝑤𝑤’) • Normalized cross-correlation:
• Advantage: Invariant to intensity differences: Invariant to affine intensity transformation 𝐼𝐼′ = 𝛼𝛼𝐼𝐼 + 𝜇𝜇
![Page 45: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/45.jpg)
Cross-correlation of neighbourhood regions
epipolar line
• left and right windows encoded as vectors w and w’• zero-mean vectors (𝑤𝑤 − �𝑤𝑤) and (𝑤𝑤′ − �𝑤𝑤’) • Normalized cross-correlation:
• Advantage: Invariant to intensity differences: Invariant to affine intensity transformation 𝐼𝐼′ = 𝛼𝛼𝐼𝐼 + 𝜇𝜇
![Page 46: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/46.jpg)
Cross-correlation of neighbourhood regions
epipolar line
• left and right windows encoded as vectors w and w’• zero-mean vectors (𝑤𝑤 − �𝑤𝑤) and (𝑤𝑤′ − �𝑤𝑤’) • Normalized cross-correlation:
• Advantage: Invariant to intensity differences: Invariant to affine intensity transformation 𝐼𝐼′ = 𝛼𝛼𝐼𝐼 + 𝜇𝜇
![Page 47: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/47.jpg)
Correlation-based window matching
Source: Andrew Zisserman
![Page 48: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/48.jpg)
Correlation-based window matching
Source: Andrew Zisserman
![Page 49: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/49.jpg)
Correlation-based window matching
Source: Andrew Zisserman
![Page 50: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/50.jpg)
Correlation-based window matching
Source: Andrew Zisserman
![Page 51: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/51.jpg)
Correlation-based window matching
Source: Andrew Zisserman
![Page 52: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/52.jpg)
Textureless regions
Source: Andrew Zisserman Grauman
![Page 53: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/53.jpg)
Textureless regions
Source: Andrew Zisserman
Textureless regions are non-distinct; high ambiguity for matches.
Grauman
![Page 54: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/54.jpg)
Textureless regions
Source: Andrew Zisserman
Textureless regions are non-distinct; high ambiguity for matches.
Grauman
![Page 55: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/55.jpg)
Textureless regions
Source: Andrew Zisserman Grauman
![Page 56: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/56.jpg)
Textureless regions
Source: Andrew Zisserman
Textureless regions are non-distinct; high ambiguity for matches,→ wrong matches
Grauman
![Page 57: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/57.jpg)
Effect of window size
Source: Andrew Zisserman Grauman
![Page 58: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/58.jpg)
Effect of window size
Source: Andrew Zisserman Grauman
![Page 59: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/59.jpg)
Effect of window size
Source: Andrew Zisserman Grauman
![Page 60: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/60.jpg)
Problems with window matching
![Page 61: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/61.jpg)
Effect of window size
Figures from Li Zhang
Want window large enough to have sufficient intensity variation, yet small enough to contain only pixels with about the same disparity.
Grauman
![Page 62: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/62.jpg)
Effect of window size
W = 3 W = 20
Figures from Li Zhang
Want window large enough to have sufficient intensity variation, yet small enough to contain only pixels with about the same disparity.
Grauman
![Page 63: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/63.jpg)
Problems?
• Ordering• Occlusion• Foreshortening
Solutions:• Formulate Constraints• Use more than two views• Smart solutions vs. “brute force” searches with
statistics
![Page 64: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/64.jpg)
Exploiting scene constraints
![Page 65: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/65.jpg)
Additional geometric constraints for correspondence
[Faugeras, pp. 321]
• Ordering of points: Continuous surface: same order in both images.
• Is that always true?
A B C A B C
A B C
A B C
![Page 66: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/66.jpg)
The Ordering Constraint
In general the pointsare in the same orderon both epipolar lines.
![Page 67: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/67.jpg)
The Ordering Constraint
But it is not always the case..
![Page 68: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/68.jpg)
Ordering constraint
1 2 3 4,5 6 1 2,3 4 5 6
21 3 4,5 61
2,3
4
5
6
surface slice surface as a path
occlusion right
occlusion left
![Page 69: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/69.jpg)
Stereo matching
Similarity measure(SSD or NCC)
Constraints• epipolar• ordering• uniqueness• disparity limit
Trade-off• Matching cost (data)• Discontinuities (prior)
Consider all paths that satisfy the constraints
pick best using dynamic programming
![Page 70: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/70.jpg)
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar• ordering• uniqueness• disparity limit
Trade-off• Matching cost (data)• Discontinuities (prior)
Consider all paths that satisfy the constraints
pick best using dynamic programming
![Page 71: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/71.jpg)
Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.
![Page 72: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/72.jpg)
Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.
![Page 73: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/73.jpg)
Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.
![Page 74: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/74.jpg)
Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.
![Page 75: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/75.jpg)
Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.
![Page 76: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/76.jpg)
Dynamic Programming (Baker and Binford, 1981)
Find the minimum-cost path going monotonicallydown and right from the top-left corner of thegraph to its bottom-right corner.
• Nodes = matched feature points (e.g., edge points).• Arcs = matched intervals along the epipolar lines.• Arc cost = discrepancy between intervals.
![Page 77: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/77.jpg)
The Ordering Constraint
In general the pointsare in the same orderon both epipolar lines.
![Page 78: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/78.jpg)
The Ordering Constraint
But it is not always the case..
![Page 79: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/79.jpg)
Forbidden Zone
m1 m2
M
![Page 80: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/80.jpg)
Forbidden Zone
m1 m2
M
N
n1 n2
![Page 81: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/81.jpg)
Forbidden Zone of M:
Violation of ordering constraints
Forbidden Zone
m1 m2
M
N
n1 n2
![Page 82: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/82.jpg)
Forbidden Zone of M:
Violation of ordering constraints
Forbidden Zone
m1 m2
M
N
n1 n2
Practical applications: – Object bulges out: ok– In general: ordering across
whole image is not reliable feature
– Use ordering constraints for neighbors of M within small neighborhood only
![Page 83: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/83.jpg)
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
![Page 84: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/84.jpg)
Hierarchical stereo matchingD
owns
ampl
ing
(Gau
ssia
n py
ram
id)
Dis
pari
ty p
ropa
gati
on
Allows faster computation
Deals with large disparity ranges
![Page 85: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/85.jpg)
Dynamic Programming (Ohta and Kanade, 1985)
Reprinted from “Stereo by Intra- and Intet-Scanline Search,” by Y. Ohta and T. Kanade, IEEE Trans. on Pattern Analysis and MachineIntelligence, 7(2):139-154 (1985). 1985 IEEE.
![Page 86: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/86.jpg)
Real-time stereo on graphics hardware
• Computes Sum-of-Square-Differences• Hardware mip-map generation used to aggregate results over
support region• Trade-off between small and large support window
Ruigang Yang and Marc Pollefeys, UNC
140M disparity hypothesis/sec on Radeon 9700proe.g. 512x512x20disparities at 30Hz
Shape of a kernel for summing up 6 levels
![Page 87: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/87.jpg)
Stereo results
Ground truthScene
– Data from University of Tsukuba– Similar results on other images without ground
truth
![Page 88: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/88.jpg)
![Page 89: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/89.jpg)
Results with window correlation
Window-based matching(best window size)
Ground truth
![Page 90: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/90.jpg)
Results with better method
State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.Ground truth
![Page 91: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/91.jpg)
Material I
• http://vision.middlebury.edu/stereo/
• (online stereo pairs and truth (depth maps)• Stereo correspondence software: e.g.
http://vision.middlebury.edu/stereo/data/scenes2001/data/imagehtml/tsukuba.html
• CVonline compendium: http://homepages.inf.ed.ac.uk/rbf/CVonline/
![Page 92: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/92.jpg)
Material II
• Epipolar Geometry, Rectification: • http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/FUSIELLO2/re
ctif_cvol.html• and:
http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/OWENS/LECT11/node11.html
• Stereo:• http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/OWENS/LECT
11/lect11.html• 3D Reconstruction:• http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/OWENS/LECT
11/node8.html
![Page 93: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/93.jpg)
Additional Materials
![Page 94: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/94.jpg)
Problem: Foreshortening
![Page 95: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/95.jpg)
Why is cross-correlation such a poor measure in the second case?
1. The neighbourhood region does not have a “distinctive” spatial intensity distribution
2. Foreshortening effects
![Page 96: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/96.jpg)
Why is cross-correlation such a poor measure in the second case?
1. The neighbourhood region does not have a “distinctive” spatial intensity distribution
2. Foreshortening effects
![Page 97: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/97.jpg)
Why is cross-correlation such a poor measure in the second case?
1. The neighbourhood region does not have a “distinctive” spatial intensity distribution
2. Foreshortening effects
![Page 98: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/98.jpg)
Why is cross-correlation such a poor measure in the second case?
1. The neighbourhood region does not have a “distinctive” spatial intensity distribution
2. Foreshortening effects
fronto-parallel surfaceimaged length the same
![Page 99: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/99.jpg)
Why is cross-correlation such a poor measure in the second case?
1. The neighbourhood region does not have a “distinctive” spatial intensity distribution
2. Foreshortening effects
fronto-parallel surfaceimaged length the same
![Page 100: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/100.jpg)
Why is cross-correlation such a poor measure in the second case?
1. The neighbourhood region does not have a “distinctive” spatial intensity distribution
2. Foreshortening effects
fronto-parallel surfaceimaged length the same
![Page 101: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/101.jpg)
Why is cross-correlation such a poor measure in the second case?
1. The neighbourhood region does not have a “distinctive” spatial intensity distribution
2. Foreshortening effects
fronto-parallel surfaceimaged length the same
slanting surfaceimaged lengths differ
![Page 102: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/102.jpg)
Three Views
The third eye can be used for verification..
Demo epipolar geometry
![Page 103: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/103.jpg)
Three Views
The third eye can be used for verification..
Demo epipolar geometry
![Page 104: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/104.jpg)
Three Views
The third eye can be used for verification..
Demo epipolar geometry
![Page 105: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/105.jpg)
More Views (Okutami and Kanade, 1993)
New book: Ch7.6 p. 215: Pick a reference image, and slide the corresponding window along the corresponding epipolarlines of all other images, using inverse depth (𝑍𝑍−1) relative to the first image as the search parameter.
Use the sum of correlation scores to rank matches: SSD used as global evaluation function: Find 𝑍𝑍−1 that minimizes SSD.
Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on PatternAnalysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.
![Page 106: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/106.jpg)
Multi-camera configurations
Okutami and Kanade
(illustration from Pascal Fua)
![Page 107: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/107.jpg)
I1 I2 I10
Reprinted from “A Multiple-Baseline Stereo System,” by M. Okutami and T. Kanade, IEEE Trans. on PatternAnalysis and Machine Intelligence, 15(4):353-363 (1993). \copyright 1993 IEEE.
![Page 108: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/108.jpg)
![Page 109: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/109.jpg)
Normalized cross correlation
Source: Andrew Zisserman
![Page 110: Multi-View Geometry: Find Corresponding Points …engineering.nyu.edu/~gerig/CS-GY-6643-S2016/Materials/CS...Multi-View Geometry: Find Corresponding Points (New book: Ch7.4, 7.5, 7.6](https://reader036.fdocuments.us/reader036/viewer/2022081404/5f05ffc17e708231d415c914/html5/thumbnails/110.jpg)
Aggregation window sizesSmall windows • disparities similar• more ambiguities• accurate when correct
Large windows • larger disp. variation• more discriminant• often more robust• use shiftable windows to deal
with discontinuities
(Illustration from Pascal Fua)