Slide 1

GrabCut Interactive Image (and Stereo) Segmentation Joon Jae Lee Keimyung University

Slide 2

Characteristics Improved from graph cut Improved from graph cut Use Gaussian Mixture Model (GMM) for clustering in color space Use Gaussian Mixture Model (GMM) for clustering in color space Iterative optimization Iterative optimization User interaction greatly reduced compared User interaction greatly reduced compared with other methods (Magic Wand, Intelligent with other methods (Magic Wand, Intelligent Scissors) Scissors)

Slide 3

Photomontage Photomontage GrabCut Interactive Foreground Extraction 1

Slide 4

Problem Problem GrabCut Interactive Foreground Extraction 3 Fast & Accurate ?

Slide 5

What GrabCut does What GrabCut does User Input Result Magic Wand (198?) Intelligent Scissors Mortensen and Barrett (1995) GrabCut Regions Boundary Regions & Boundary GrabCut Interactive Foreground Extraction 4

Slide 6

Gaussian Mixture Model The probability of a pixel vector is represented The probability of a pixel vector is represented as: as:

Slide 7

Two sets of GMM, one for background, one Two sets of GMM, one for background, one for unknown region for unknown region Gaussian Mixture Model

Slide 8

Algorithm For each pixel assign a value For each pixel assign a value = 0 for background and = 1 for foreground = 0 for background and = 1 for foreground Use graph cut to minimize a Gibbs energy: Use graph cut to minimize a Gibbs energy: : background / foreground label : background / foreground label : GMM parameters : GMM parameters k: GMM labeling k: GMM labeling z: pixel value z: pixel value

Slide 9

Iterative minimization Initialize background pixel to = 0 and unknown region (draw box) to = 1. Initialize background pixel to = 0 and unknown region (draw box) to = 1. Initialize two sets of GMMs. Initialize two sets of GMMs. 1. Assign GMM labels to each pixel. (Which set of GMM is determined by ) 1. Assign GMM labels to each pixel. (Which set of GMM is determined by ) 2. Graph cut minimize ( optimized, GMM label changed to corresponding set of GMM) 2. Graph cut minimize ( optimized, GMM label changed to corresponding set of GMM) 3. Update GMM parameters 3. Update GMM parameters 4. Repeat from step 1 until convergence 4. Repeat from step 1 until convergence

Slide 10

Graph Cuts - Boykov and Jolly (2001) Graph Cuts - Boykov and Jolly (2001) GrabCut Interactive Foreground Extraction 7 Image Min Cut Min Cut Cut: separating source and sink; Energy: collection of edges Min Cut: Global minimal enegry in polynomial time Foreground (source) Background (sink)

Slide 11

1 2 34 Iterated Graph Cuts Iterated Graph Cuts GrabCut Interactive Foreground Extraction 9 Energy after each IterationResult Guaranteed to converge

Slide 12

Colour Model Colour Model Gaussian Mixture Model (typically 5-8 components) Foreground & Background Background Foreground Background G GrabCut Interactive Foreground Extraction 10 R G R Iterated graph cut

Slide 13

Coherence Model Coherence Model An object is a coherent set of pixels: Error (%) over training set: 25 How do we choose ? 25

Slide 14

A Gaussian MRF is not a realistic texture model Gaussian? Gaussian! Real Image synthetic GMRF Parameter Learning - Problems GrabCut Interactive Foreground Extraction 13

Slide 15

Moderately simple examples Moderately simple examples GrabCut completes automatically GrabCut Interactive Foreground Extraction 14

Slide 16

Difficult Examples Difficult Examples Camouflage & Low Contrast No telepathy Fine structure Initial Rectangle Initial Result GrabCut Interactive Foreground Extraction 15

Slide 17

Evaluation Labelled Database Available online: http://research.microsoft.com/vision/cambridge/segmentation/ GrabCut Interactive Foreground Extraction 16

Slide 18

Comparison Comparison GrabCut Boykov and Jolly (2001) Error Rate: 0.72% Error Rate: 1.87%Error Rate: 1.81%Error Rate: 1.32%Error Rate: 1.25%Error Rate: 0.72% GrabCut Interactive Foreground Extraction 17 User Input Result

Slide 19

Comparison Comparison Trimap Boykov and Jolly Error Rate: 1.36% Input Image Ground Truth Bimap GrabCut Error Rate: 2.13% GrabCut Interactive Foreground Extraction 18 Error rate - modestly increase User Interactions - considerable reduced

Slide 20

Results Parameter Learning GrabCut Interactive Foreground Extraction 19

Slide 21

Comparison Comparison Magic Wand (198?) Intelligent Scissors Mortensen and Barrett (1995) GrabCut Rother et al. (2004) Graph Cuts Boykov and Jolly (2001) LazySnapping Li et al. (2004) GrabCut Interactive Foreground Extraction 20