Computer Vision, Robert Pless our goal is to understand the process of stereo vision. Last time, we...

Computer Vision, Robert Pless

• our goal is to understand the process of stereo vision.

• Last time, we studied the “Essential” Matrix, which describe where a point in one image may appear in a second image.


Rectification

Need to rotate each camera so the epipole is along the y-axis. (0,+/-1,0)Is this unique?


Stereo Image Rectification


• reproject image planes onto a common• plane parallel to the line between optical centers• pixel motion is horizontal after this transformation C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision.

IEEE Conf. Computer Vision and Pattern Recognition, 1999.

Stereo Image Rectification


Stereo Rectification


Rectification


Basic Stereo Algorithm

For each epipolar line

For each pixel in the left image• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost

Improvement: match windows• This should look familar...

• Correlation, Sum of Squared Difference (SSD), etc.


Then, finally,

OOll OOrr

PP

ppll pprr

bb

ZZ

xxll xxrr

ff

b b is the stereo baselineis the stereo baselined d measures the difference in retinal position between corresponding measures the difference in retinal position between corresponding pointspoints

Then given Z, we can compute X and Y.

Disparity:


Using these constraints we can use matching for stereo

For each epipolar lineFor each pixel in the left image

• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost

• This will never work, so:

Improvement: match windows


Correspondence: Photometric constraint

• Same world point has same intensity in both images.– Lambertian fronto-parallel– Issues:

• Noise• Specularity• Foreshortening


Comparing Windows: ==??

ff gg

MostMostpopularpopular

For each window, match to closest window on epipolar line in other image.


Window size

W = 3 W = 20

Better results with adaptive window• T. Kanade and M. Okutomi, A Stereo Matching

Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.

• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998

• Effect of window size


Stereo results

Ground truthScene

– Data from University of Tsukuba


Results with window correlation

Window-based matching(best window size)

Ground truth


Results with better method

State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth


Stereo Results

Trinocular stereo system available from Point Gray Research for $5K (circa ’97)


Stereo Example

input image (1 of 2) [Szeliski & Kang ‘95]

depth map 3D rendering


Stereo Example

left image right image depth map

H. Tao et al. “Global matching criterion and color segmentation based stereo”


Stereo Example

H. Tao et al. “Global matching criterion and color segmentation based stereo”


What are these additional constraints?

corresponding points along corresponding

epipolar lines


Order !

Once the epipolar geometry is known, we can increase constraints on potential correspondences further...


epipolar lines 3d points giving rise to those images


Order !

Out-of-order points indicate “seeing around” an object, and generally doesn’t occur ...


epipolar lines 3d points giving rise to those imagesCan you see the dynamic

programming problem here?


Correspondence

• It is fundamentally ambiguous, even with stereo constraints

Ordering constraint… …and its failure


Search Over Correspondences

Three cases:– Sequential – cost of match– Occluded – cost of no match– Disoccluded – cost of no match

Left scanline

Right scanline

Occluded Pixels

Dis-occluded Pixels


Stereo Matching with Dynamic Programming

Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Start

End


Stereo Matching with Dynamic Programming

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal


Computing Correspondence

• Another approach is to match edges rather than windows of pixels:

• Which method is better?– Edges tend to fail in dense texture (outdoors)– Correlation tends to fail in smooth featureless areas


Computing Correspondences

• Both methods fail for smooth surfaces

• There is currently no good solution to the correspondence problem


History and Philosophy of human stereo

• Features vs. Pixels?– Do we extract features prior to matching?

Julesz-style Random Dot Stereogram


Human abilities


Assignment: Stereo In or Stereo Out (2 options)

1.Write a program that takes as input a rectified stereo pair, and creates the 3D model of the scene.

1. Show your program output on:http://svs.gsfc.nasa.gov/vis/a000000/

a003500/a003531/

2.Write a program that takes as input a depth map, and creates an auto-stereogram that gives that depth map.

1. Google search for “depth map”

Computer Vision, Robert Pless our goal is to understand the process of stereo vision. Last time, we...

Documents

Transcript of Computer Vision, Robert Pless our goal is to understand the process of stereo vision. Last time, we...