The Layout Consistent Random Fieldfor detecting and segmenting occluded objects

CVPR, June 2006

John Winn

Jamie

Shotton

LayoutCRF contributions

Detection and segmentation

Handles occlusion and deformation

Multiple objects simultaneously

Multiple classes

Related work

Layout consistency

Layout Consistent Random Field

Results

Roadmap

Related work: constellation models

X

[Crandall et al. ECCV 2006]

[Fergus et al. CVPR 2003]

[Leibe et al. ECCV 2004][Kumar et al. CVPR 2005] …

Related work: constellation models

[Crandall et al. ECCV 2006]

[Fergus et al. CVPR 2003]

[Leibe et al. ECCV 2004][Kumar et al. CVPR 2005] …

X

XX

X

Related work: windowed detectors

[Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…

Localised features

Classifier Car

Sliding window

Related work: windowed detectors

Localised features

Classifier Car? Wall?

[Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…

Sliding window

Related work: multiclass segmentation

[Tu et al. CVPR 2003][He et al. CVPR 2004]

tree

car

road

building

Doesn’t exploit layout of parts – can’t identify object instances

TextonBoost [Shotton et al. ECCV 2006]

Related work

Layout consistency

Layout Consistent Random Field

Results

Roadmap

Dense part labelling

Automatic per-pixel labelling based on a grid of parts

Part labels (color-coded)

Dense part labelling

Background label

Part labels (color-coded)

Patch-based part detector

[Lepetit et al. CVPR 2005]

Decision forest classifier

Features are differences of pixel intensities

Classifier

Decision trees

Extremely efficient at both training and test time. e.g. takes 2ms to apply to 160x120 image using difference of pixel intensities.

Improved performance with multiple decision trees (random forest).

Performs as well as boosting with shared features, but can process much more data in the same time.

Patch-based part detector

Colors show posterior over part labels – part detectors are noisy!

Part color key

Layout consistency

Layout consistency

Layout consistency

(8,3) (9,3)(7,3)

(8,2) (9,2)(7,2)

(8,4) (9,4)(7,4)

Neighboring pixels

(p,q) ?

Layout consistency

(8,3) (9,3)(7,3)

(8,2) (9,2)(7,2)

(8,4) (9,4)(7,4)

Neighboring pixels

(p,q)

(p+1,q)

(p,q)

(p+1,q+1)

(p+1,q-1)

Allows fordeformation

/rotation

Layoutconsist

ent

Layout consistency

(8,3) (9,3)(7,3)

(8,2) (9,2)(7,2)

(8,4) (9,4)(7,4)

Neighboring pixels

(p,q)

? (p,q+1)(p,q) (p+1,q

+1)(p-

1,q+1)

Layoutconsist

ent

Occlusions

One object instance occludes another

‘Background’ occludes object

Object occludes background (object edge)

Not layout consistent = occlusion (or invalid)

Effect of layout consistency

Input image

With layout consistency

Part detector output

Layout consistent regions

Related work

Layout consistency

Layout Consistent Random Field

Results

Roadmap

Layout Consistent Random Field

Part detectorPart labels h

Image I

Layout Consistent Random Field

Part labels h

Layout consistency

Image I

Part detector

Layout Consistent Random Field

Parameters θ’={ βbg , βoe , βco , βiif , e0 , γ }

(set by hand)

Layout consistency

Part detector

Edge weight

Proposed

labelling

Inference of MAP labellingGraph cuts with customised alpha-expansion move

[Boykov and Jolly, ICCV 2001]

Part labels h

Inference of MAP labelling

[Boykov and Jolly, ICCV 2001]

Graph cuts with customised alpha-expansion move

Proposed

labelling

Part labels h

Inference of MAP labelling

[Boykov and Jolly, ICCV 2001]

Expansion move not accepted

Graph cuts with customised alpha-expansion move

Proposed

labelling

Part labels h

Inference of MAP labelling

[Boykov and Jolly, ICCV 2001]

Graph cuts with customised alpha-expansion move

Proposed

labelling

Part labels h

Example inference

Decision tree re-learningPart-labels are inferred (constrained by known mask) and decision forest re-trained

Limitation of layout consistency Allows arbitrary stretching/scaling

Part labels h

Global layout

Global layout

Instance T1

InstanceT2

Global layout constraint is (weak) star-shaped constellation model

Constrains part locationsrelative to centroid

Allows competition between different object instances

Image I

Example with global consistency

Input image

Layout consistent regions

Instance labelling

T1

T2

T3 T1

T2

Related work

Layout consistency

Layout Consistent Random Field

Results

Roadmap

UIUC car database

Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)

UIUC car database

Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)

UIUC car database: detection

Results refer to detection of unoccluded cars only.

Detecting heavily occluded faces Caltech face database with artificial occlusions AR face database with real occlusions

Stability of part labelling

Part color key

Multi-class detection

Can extend to multiple classes with different numbers of part labels for each class

Example: building has multiple parts, other classes have one

Summary + future directionsSummary:LayoutCRF achieves multi-class detection and segmentation of occluded, deformable objects

Future directions: Extend to multiple viewpoints and multiple

scales Share parts between classes Incorporate object context (‘car above road’) Incorporate geometric cues

Thank you

jwinn@microsoft.com

http://johnwinn.org/