Geo-consistency for Wide Multi-Camera StereoMarc-Antoine Drouin, Martin Trudeau and Sebastien Roy

Departement d’Informatique et recherche operationnelle, Universite de Montreal, Canadaemail: {drouim,trudeaum,roys}@iro.umontreal.ca

Problem: 3D Reconstruction of a scene

from a reference camera using multiple

supporting ones

For each pixel p we need to choose

•a depth f(p)

•a mask of used cameras g(p)

so as to minimize the energy

E(f, g) =∑

p∈Pe(p, f(p), g(p))

+smoothing.

Handling Visibility

Exact wheng(p) = V (p|f(p), f)

(the real visibility)

•Long range interaction ⇒ en-ergy minimization is hard

Heuristic wheng(p) = arg minm∈M e(p, f(p), m)

(M is a set of plausible masks)

•Photo-consistency �⇒ correctvisibility

Geo-consistency

Defined as

g(p) ≤ V (p|f(p), f)

•Occluded cameras are not used

•Visible cameras are not alwaysused

Justified by Nakamura96:

•Using an occluded camera ⇒important artifacts

•Not using a visible camera ⇒no significant artifacts

Bias in Border Localization

Closest objects are enlarged by stan-dard algorithms.

Supportingcamera

Referencecamera

GT occludeesGT occluders

DM occludeesDM occluders

1 2 3 4 5

•Large number of occluders indepth map are occludees accord-ing to ground truth (zone 3)

•Visibility from initial depth mapis useless to iteratively improve thesolution

Pseudo-Visibility

It compensates for the bias by label-ing both occluders and occludees asinvisible.

•Preserves Geo-consistency

•Computed using rendering tech-niques (the depth map is repre-sented as a continuous mesh)

Algorithm: Overview

• Initialize with all cameras visiblein each mask

•Compute depth map

•Update pseudo-visibility masksusing depth map

• Iterate until convergence

depth map f

initial masks g

stereo matcher

Pseudo-Visibility

final depth map f

final masks g

masks changed ?

new masks g

History

Yes

No

stereo matcher

Ordering Constraint

It is respected when the order inwhich 2 objects are encounteredalong an epipolar line does notchange.

•Not always true

•Continuous mesh ⇒ordering constraint is respected

2

1

2 1 2 1

2

1

12 21

History

Once a camera is removed, it isnever used again.

•Guaranteed convergence to aGeo-consistent solution that re-spects the ordering constraint

Results:Middlebury

An error is a difference greater than1 label from the ground truth.

Middlebury sequencealgorithmsbarn1 barn2 bull postervenussawtoothBoykov99 3.5 % 3.1 % 0.7 % 3.7 % 3.4 % 3.3%Ours 0.8 %0.6 %0.4 %1.1 %2.4 % 1.1 %Kang01 1.4 % 1.5 % 0.9 %1.1 %4.0 % 1.5%Drouin05 0.7 %3.9 % 0.8 % 4.0 % 5.3% 1.0 %

Results:Baseline Test

Depth maps recovered for the sameviewpoint with different baselinesshould be identical.

•baseline ↑ ⇒ occlusion ↑

0 2 4 6 8 10 12 14%

Boykov99Sanfourche04Kolmogorov02Sanfourche04

Ours 1x vs 2x2x vs 3x3x vs 4x1x vs 4x

% differing pixels

Ours (3x)Reference

Kang01 (3x) Kolmogorov02 (3x)

Conclusion

• New framework to model occlu-sion in stereo by introducing Geo-consistency

•Occlusion modeling is added tostandard stereo algorithms

Future Work

•Better handling of regions break-ing the ordering constraint

•Extending the framework to vol-umetric reconstruction