The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006...
-
Upload
jahiem-bathe -
Category
Documents
-
view
214 -
download
0
Transcript of The Layout Consistent Random Field for detecting and segmenting occluded objects CVPR, June 2006...
The Layout Consistent Random Fieldfor detecting and segmenting occluded objects
CVPR, June 2006
John Winn
Jamie
Shotton
LayoutCRF contributions
Detection and segmentation
Handles occlusion and deformation
Multiple objects simultaneously
Multiple classes
Related work
Layout consistency
Layout Consistent Random Field
Results
Roadmap
Related work: constellation models
X
[Crandall et al. ECCV 2006]
[Fergus et al. CVPR 2003]
[Leibe et al. ECCV 2004][Kumar et al. CVPR 2005] …
Related work: constellation models
[Crandall et al. ECCV 2006]
[Fergus et al. CVPR 2003]
[Leibe et al. ECCV 2004][Kumar et al. CVPR 2005] …
X
XX
X
Related work: windowed detectors
[Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…
Localised features
Classifier Car
Sliding window
Related work: windowed detectors
Localised features
Classifier Car? Wall?
[Viola and Jones CVPR 2001] [Shotton et al. ICCV 2005]…
Sliding window
Related work: multiclass segmentation
[Tu et al. CVPR 2003][He et al. CVPR 2004]
tree
car
road
building
Doesn’t exploit layout of parts – can’t identify object instances
TextonBoost [Shotton et al. ECCV 2006]
Related work
Layout consistency
Layout Consistent Random Field
Results
Roadmap
Dense part labelling
Automatic per-pixel labelling based on a grid of parts
Part labels (color-coded)
Dense part labelling
Background label
Part labels (color-coded)
Patch-based part detector
[Lepetit et al. CVPR 2005]
Decision forest classifier
Features are differences of pixel intensities
Classifier
Decision trees
Extremely efficient at both training and test time. e.g. takes 2ms to apply to 160x120 image using difference of pixel intensities.
Improved performance with multiple decision trees (random forest).
Performs as well as boosting with shared features, but can process much more data in the same time.
Patch-based part detector
Colors show posterior over part labels – part detectors are noisy!
Part color key
Layout consistency
Layout consistency
Layout consistency
(8,3) (9,3)(7,3)
(8,2) (9,2)(7,2)
(8,4) (9,4)(7,4)
Neighboring pixels
(p,q) ?
Layout consistency
(8,3) (9,3)(7,3)
(8,2) (9,2)(7,2)
(8,4) (9,4)(7,4)
Neighboring pixels
(p,q)
(p+1,q)
(p,q)
(p+1,q+1)
(p+1,q-1)
Allows fordeformation
/rotation
Layoutconsist
ent
Layout consistency
(8,3) (9,3)(7,3)
(8,2) (9,2)(7,2)
(8,4) (9,4)(7,4)
Neighboring pixels
(p,q)
? (p,q+1)(p,q) (p+1,q
+1)(p-
1,q+1)
Layoutconsist
ent
Occlusions
One object instance occludes another
‘Background’ occludes object
Object occludes background (object edge)
Not layout consistent = occlusion (or invalid)
Effect of layout consistency
Input image
With layout consistency
Part detector output
Layout consistent regions
Related work
Layout consistency
Layout Consistent Random Field
Results
Roadmap
Layout Consistent Random Field
Part detectorPart labels h
Image I
Layout Consistent Random Field
Part labels h
Layout consistency
Image I
Part detector
Layout Consistent Random Field
Parameters θ’={ βbg , βoe , βco , βiif , e0 , γ }
(set by hand)
Layout consistency
Part detector
Edge weight
Proposed
labelling
Inference of MAP labellingGraph cuts with customised alpha-expansion move
[Boykov and Jolly, ICCV 2001]
Part labels h
Inference of MAP labelling
[Boykov and Jolly, ICCV 2001]
Graph cuts with customised alpha-expansion move
Proposed
labelling
Part labels h
Inference of MAP labelling
[Boykov and Jolly, ICCV 2001]
Expansion move not accepted
Graph cuts with customised alpha-expansion move
Proposed
labelling
Part labels h
Inference of MAP labelling
[Boykov and Jolly, ICCV 2001]
Graph cuts with customised alpha-expansion move
Proposed
labelling
Part labels h
Example inference
Decision tree re-learningPart-labels are inferred (constrained by known mask) and decision forest re-trained
Limitation of layout consistency Allows arbitrary stretching/scaling
Part labels h
Global layout
Global layout
Instance T1
InstanceT2
Global layout constraint is (weak) star-shaped constellation model
Constrains part locationsrelative to centroid
Allows competition between different object instances
Image I
Example with global consistency
Input image
Layout consistent regions
Instance labelling
T1
T2
T3 T1
T2
Related work
Layout consistency
Layout Consistent Random Field
Results
Roadmap
UIUC car database
Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)
UIUC car database
Segmentation accuracy: 96.5% pixels correct (assessed on 20 randomly selected, hand-labelled images)
UIUC car database: detection
Results refer to detection of unoccluded cars only.
Detecting heavily occluded faces Caltech face database with artificial occlusions AR face database with real occlusions
Stability of part labelling
Part color key
Multi-class detection
Can extend to multiple classes with different numbers of part labels for each class
Example: building has multiple parts, other classes have one
Summary + future directionsSummary:LayoutCRF achieves multi-class detection and segmentation of occluded, deformable objects
Future directions: Extend to multiple viewpoints and multiple
scales Share parts between classes Incorporate object context (‘car above road’) Incorporate geometric cues