Bi-layer segmentationBi-layer segmentation of binocular stereo video of binocular stereo video

V. Kolmogorov, A. Criminisi, A. Blake, G. Cross, C. Rother

Microsoft Research, Cambridge, UKComputer Vision Group

ProblemProblem

• Two layers (Bg and Fg)– Scenarios: video-conferencing, live bluescreening without bluescreen

• Two cameras• Background may not be static• Goal: accurately segment foreground object in real-time

– Applications: background substitution, ...

Left input Right input

Sources of informationSources of information

• Stereo– Foreground object has larger disparity

• Colour– Background and foreground have distinct colour

distributions

• Contrast– There is image gradient at Bg/Fg transition

• Spatial coherence– MRF model

Previous workPrevious work

• Colour/Contrast (+2D coherence)– Graph cuts

(Boykov et al. ’01, Rother et al. ’04)

• Stereo (+1D coherence)– Dynamic programming

(Ohta et al. ’85, Cox et al. ’96, Criminisi et al. ’03)

Left input

Fusing colour/contrast and stereoFusing colour/contrast and stereo

• Colour and stereo complement each other• Result from fusion:

Results from our technique

• Ideally:

segmentation (Bg/Fg/Occ)

disparity

Fusing colour/contrast and stereoFusing colour/contrast and stereo

data

Construct p(x , d | z) Prior p(x , d) –2D MRF model

Marginalise d out to get p(x | z)

Compute x as MAP configuration Intractable!

Approximations

• Simplify the model to get real-time performance• Two different approaches:

– Layered Dynamic Programming (LDP)– Layered Graph Cut (LGC)

• Probabilistic formulation– Parameters can mostly be set automatically

• Very similar error statistics– Consistently better than colour/contrast or stereo alone

Layered Dynamic Programming (LDP)Layered Dynamic Programming (LDP)

• Approximation: in the prior neglect coupling between scanlines

• 1D MRF model (Markov chain)

• Compute path from (0,0) to (N,N) and label {Fg,Bg,Occ} for each vertex

Left

Rig

ht

Disparity

Layered Graph Cut (LGC)Layered Graph Cut (LGC)

• Approximation: in the prior neglect conditioning of disparity dp on disparities of neighbours

– conditioned only on segmentation label xp{Bg,Fg,Occ}

• Marginalise disparities out• Energy minimisation problem with 3 labels• Solve it using 2 graph cut computations

– Approx. 20 frames per second (320 x 240, 3GHz)

Setting parametersSetting parameters

• Two approaches:– Generative: from physics (e.g. from average width of

occluded regions)– Discriminative: minimize error rates

• Consistent results! – See technical report 2005 (

http://research.microsoft.com/vision/cambridge/i2i/)

Experiments: ground truth dataExperiments: ground truth data

Ground-Truth segmentation at http://research.microsoft.com/vision/cambridge/i2i/

• 19 calibrated stereo sequences• 6 with ground truth segmentation

– Every 5th or 10th frame– Pixels marked as “Bg”, “Fg” or “Unknown”

Original Ground-truth segmentation

Accuracy Accuracy ofof segmentation segmentation

LGC Segmentation

Accuracy of segmentationAccuracy of segmentation

Accuracy of segmentationAccuracy of segmentation

ConclusionConclusion

• Two algorithms based on different approximations– Fuse colour/constrast, stereo, and spatial coherence– Probabilistic formulation– Capable of real-time performance

• Similar error statistics– Consistently better than state-of-the art techniques

• Different characteristics– LDP: Parallelisable (scanlines processed independently)– LGC: Marginalisation could be done on GPU

MSN: i2i cambridge

Segmentation errors (LGC)

Segmentation errors (LGC)