Computer Vision, Robert Pless

• our goal is to understand the process of stereo vision.

• Last time, we studied the “Essential” Matrix, which describe where a point in one image may appear in a second image.

Computer Vision, Robert Pless

Rectification

Need to rotate each camera so the epipole is along the y-axis. (0,+/-1,0)Is this unique?

Computer Vision, Robert Pless

Stereo Image Rectification

Computer Vision, Robert Pless

Computer Vision, Robert Pless

• reproject image planes onto a common• plane parallel to the line between optical centers• pixel motion is horizontal after this transformation C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision.

IEEE Conf. Computer Vision and Pattern Recognition, 1999.

Stereo Image Rectification

Computer Vision, Robert Pless

Stereo Rectification

Computer Vision, Robert Pless

Rectification

Computer Vision, Robert Pless

Basic Stereo Algorithm

For each epipolar line

For each pixel in the left image• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost

Improvement: match windows• This should look familar...

• Correlation, Sum of Squared Difference (SSD), etc.

Computer Vision, Robert Pless

Then, finally,

OOll OOrr

PP

ppll pprr

bb

ZZ

xxll xxrr

ff

b b is the stereo baselineis the stereo baselined d measures the difference in retinal position between corresponding measures the difference in retinal position between corresponding pointspoints

Then given Z, we can compute X and Y.

Disparity:

Computer Vision, Robert Pless

Using these constraints we can use matching for stereo

For each epipolar lineFor each pixel in the left image

• compare with every pixel on same epipolar line in right image

• pick pixel with minimum match cost

• This will never work, so:

Improvement: match windows

Computer Vision, Robert Pless

Correspondence: Photometric constraint

• Same world point has same intensity in both images.– Lambertian fronto-parallel– Issues:

• Noise• Specularity• Foreshortening

Computer Vision, Robert Pless

Comparing Windows: ==??

ff gg

MostMostpopularpopular

For each window, match to closest window on epipolar line in other image.

Computer Vision, Robert Pless

Window size

W = 3 W = 20

Better results with adaptive window• T. Kanade and M. Okutomi, A Stereo Matching

Algorithm with an Adaptive Window: Theory and Experiment,, Proc. International Conference on Robotics and Automation, 1991.

• D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998

• Effect of window size

Computer Vision, Robert Pless

Stereo results

Ground truthScene

– Data from University of Tsukuba

Computer Vision, Robert Pless

Results with window correlation

Window-based matching(best window size)

Ground truth

Computer Vision, Robert Pless

Results with better method

State of the art methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,

International Conference on Computer Vision, September 1999.

Ground truth

Computer Vision, Robert Pless

Stereo Results

Trinocular stereo system available from Point Gray Research for $5K (circa ’97)

Computer Vision, Robert Pless

Stereo Example

input image (1 of 2) [Szeliski & Kang ‘95]

depth map 3D rendering

Computer Vision, Robert Pless

Stereo Example

left image right image depth map

H. Tao et al. “Global matching criterion and color segmentation based stereo”

Computer Vision, Robert Pless

Stereo Example

H. Tao et al. “Global matching criterion and color segmentation based stereo”

Computer Vision, Robert Pless

What are these additional constraints?

corresponding points along corresponding

epipolar lines

Computer Vision, Robert Pless

Order !

Once the epipolar geometry is known, we can increase constraints on potential correspondences further...

corresponding points along corresponding

epipolar lines 3d points giving rise to those images

Computer Vision, Robert Pless

Order !

Out-of-order points indicate “seeing around” an object, and generally doesn’t occur ...

corresponding points along corresponding

epipolar lines 3d points giving rise to those imagesCan you see the dynamic

programming problem here?

Computer Vision, Robert Pless

Correspondence

• It is fundamentally ambiguous, even with stereo constraints

Ordering constraint… …and its failure

Computer Vision, Robert Pless

Search Over Correspondences

Three cases:– Sequential – cost of match– Occluded – cost of no match– Disoccluded – cost of no match

Left scanline

Right scanline

Occluded Pixels

Dis-occluded Pixels

Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming

Dynamic programming yields the optimal path through grid. This is the best set of matches that satisfy the ordering constraint

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Start

End

Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Computer Vision, Robert Pless

Stereo Matching with Dynamic Programming

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Computer Vision, Robert Pless

Computing Correspondence

• Another approach is to match edges rather than windows of pixels:

• Which method is better?– Edges tend to fail in dense texture (outdoors)– Correlation tends to fail in smooth featureless areas

Computer Vision, Robert Pless

Computing Correspondences

• Both methods fail for smooth surfaces

• There is currently no good solution to the correspondence problem

Computer Vision, Robert Pless

History and Philosophy of human stereo

• Features vs. Pixels?– Do we extract features prior to matching?

Julesz-style Random Dot Stereogram

Computer Vision, Robert Pless

Human abilities

Computer Vision, Robert Pless

Human abilities

Computer Vision, Robert Pless

Assignment: Stereo In or Stereo Out (2 options)

1.Write a program that takes as input a rectified stereo pair, and creates the 3D model of the scene.

1. Show your program output on:http://svs.gsfc.nasa.gov/vis/a000000/

a003500/a003531/

2.Write a program that takes as input a depth map, and creates an auto-stereogram that gives that depth map.

1. Google search for “depth map”