Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris...
-
Upload
lorin-sherman -
Category
Documents
-
view
222 -
download
0
Transcript of Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris...
![Page 1: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/1.jpg)
1
Associative Hierarchical CRFs for Object Class Image Segmentation
L’ubor Ladick’y and Chris RussellOxford Brookes University
Pushmeet KohliMicrosoft Research Cambridge
International Conference on Computer Vision (ICCV) 2009
Philip H.S. TorrOxford Brookes University
![Page 2: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/2.jpg)
2
Outline• Introduction
• Random Fields for Labelling Problems
• Hierarchical CRF for Object Segmentation
• Experiments and Results
![Page 3: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/3.jpg)
3
Structure
![Page 4: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/4.jpg)
4
Introduction
![Page 5: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/5.jpg)
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
Mean Shift
![Page 6: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/6.jpg)
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
Mean Shift
![Page 7: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/7.jpg)
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
Mean Shift
![Page 8: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/8.jpg)
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
Mean Shift
![Page 9: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/9.jpg)
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
Mean Shift
![Page 10: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/10.jpg)
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Mean Shiftvector
Objective : Find the densest region
Mean Shift
![Page 11: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/11.jpg)
Distribution of identical billiard balls
Region ofinterest
Center ofmass
Objective : Find the densest region
Mean Shift
![Page 12: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/12.jpg)
12
Pixel V.S. Segment The difference of pair CRFs between based on pixels and segments
a) Based on pixels:a) No quantization errors
b) Lack of long range interactions
c) Results oversmoothed
b) Based on segmentsa) Allows long range interactions
b) Can not recover from incorrect segmentation
![Page 13: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/13.jpg)
13
Introduction Propose a novel hierarchical CRF
Integration of features derived for different quantisation levels
Propose new sophisticated potentials defined over the different levels of the quantisation hierarchy
Use a novel formulation that allows context to be incorporate at multiple levels of multiple quantisation
![Page 14: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/14.jpg)
14
Random Fields for Labelling Problems Introduce the pixel-based CRF used for formulation the
object class segmentation problem
One discrete R.V. per image pixel, each of which may take a value from the set of labels
Symbols:
Label:
R.V. :
Pixel
The set of all neighbours of the variable :
A clique c is a set of random variables
Labelling: denoted by take the value from
![Page 15: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/15.jpg)
15
CRFs
D: the set of the data, Z: the partition function, C: the set of all cliques
: the potential function of the clique ,
The energy form:
The most probable or MAP labeling :
Wrote as the sum of unary and pairwise potentials
![Page 16: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/16.jpg)
16
The Robust Model Extended by the robust potentials [KohLi et al., 2008 ]
S: the set of the segments
The pixels within the same segment are more like likely to take the same label
the form of the robust potentials:
;
The weighted version:
![Page 17: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/17.jpg)
17
-Based Hierarchical CRFs
The single auxiliary variable where c is a segment or a clique
Take the value from an extended label set
![Page 18: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/18.jpg)
18
-Based Hierarchical CRFs New cost function over
The unary potential over Y,
The pairwise potential
over Y and X
Goal:
The new energy function:
![Page 19: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/19.jpg)
19
Recursive Form
The auxiliary variables in the last layer are the input variable
The new energy function:
The recursive form:
Initial form:
![Page 20: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/20.jpg)
20
Hierarchical CRF for Object Segmentation
Describe the set of potentials in the object-class segmentation problem
Include unary potentials for both pixels and segments, pairwise potentials between pixels and segments and connective potentials between pixels and their containing segments
![Page 21: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/21.jpg)
21
Robustness to misleading segmentations
Using unsupervised segmentation algorithm may be misleading – segment may contain multiple object classed
Assigning the same label to all pixels will result in an incorrect labeling
Overcome it by using the segment quality measures [Rabinovich et at., 2009] and [Ren and Malik, 2003]
By modifying the potentials according to a quality sensitive measure for all segment c
Writing weight features based potential over c
![Page 22: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/22.jpg)
22
Potentials for object class segmentation Refer to elements of each layer as pixels, segments, and super-
segments
At the pixel level:
The unary potentials are computed using a boosted dense feature classifier [Shotton et al., 2006]
The pairwise potentials [Boykov and Jolly, 2001] , [Rother at al., 2004]:
![Page 23: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/23.jpg)
23
Potentials for object class segmentation
At the segment level:
Initially found using a fine scale mean-shift algorithm [Comaniciu and Meer, 2002]
Contain little novel local information, but strong predictors of consistency
The potentials learning at this level are uniform, due to the lack of unique features, however as they are strongly indicative of local consistency, the penalty associated with breaking them is high
To encourage neighbouring segments with similar texture to take the same label, used pairwise potentials based on the Euclidean distance if normalized histograms of colour
![Page 24: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/24.jpg)
24
Potentials for object class segmentation
At the super-segment level:
Based upon a coarse mean-shift segmentation, performed over the result of the previous segmentations
Contain significantly more internal information than their smaller children
Propose unary segment potential based on the histograms of features
![Page 25: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/25.jpg)
25
Unary Potentials From Dense Feature
Perform texture based segmentation at pixel level
Derived from TextonBoost [Shotton et al., 2006]
The features are computed on every pixel
Extend the TextonBoost by boosting classifiers defined on multiple dense feature together
Dense–feature shape filters defined by triplets: [f, t, r] where f is a feature type, t is a feature cluster, and r is a rectangular region
Feature response : Given a point i, the number of features of type f belong to the cluster t in the region r relative to the point i
![Page 26: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/26.jpg)
26
Histogram-Based Segment Unary Potentials
Defined over segments and super-segments
The distribution of dense features responses are more discriminative than any feature along
The unary potential of an auxiliary variable representing a segment is learnt by (using the normalized histograms of multiple clustered dense features) using multi-class Gentle Ada-boost[Torralba et al., 2004]
Weak classifiers: f: the normalized histogram of the feature set t: the cluster index a: threshold
![Page 27: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/27.jpg)
27
Histogram-Based Segment Unary Potentials
The segment potential: a : the response given by the Ada-boost classifier to clique c taking label l a : the truncation threshold a , and a normalizing constant
![Page 28: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/28.jpg)
28
Learning Weights for Hierarchical CRFs
Uses a coarse to fine, layer-based, local search scheme over a validation set
Introduce additional notation:
a) : the variable contained in the layer
b) : the labelling of associated with a MAP estimate
c) Determine a dominant label for each segment c, such that when , if there is no such dominant label, set a
d) The label of a clique : correspond to the dominant label of this clique(segment) in the ground truth (or ) for its containing ot be correctly labelled
![Page 29: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/29.jpg)
29
Learning Weights for Hierarchical CRFs
At each layer, seek to minimize the discrepancy between the dominant ground truth of a clique(segment), and the value of the MAP estimate
Choose parameters λ to minimize
![Page 30: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/30.jpg)
30
Algorithm
: the weighting of unary terms in the layer a : the weighting of pairwise terms in the layer a : a scalar modifier of all terms in the layer a : an arbitrary constant that controls the precision of the final assignment of
![Page 31: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/31.jpg)
31
Experiments Two data sets
1. MSRC-21 [Shotton et al., 2006]
Resolution: pixels
21 object classes
2. PASCAL VOC 2008 [Everingham et al., 2008, website]
511 training, 512 validation and 512 segmented test images
20 foreground and 1 background classes
10, 057 images for which only the bounding boxes of the objects present in the image are marked
![Page 32: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/32.jpg)
32
Results on The MSRC-21
![Page 33: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/33.jpg)
33
Results on The MSRC-21
[25]: J. Shotton et al., CVPR, 2008
[26]: J. Shotton et al., ECCV, 2006
[1]: D. Batra et al., CVPR, 2008
[25]: L. Yang et al., CVPR, 2007
![Page 34: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/34.jpg)
34
Results on The MSRC-21
[25]: J. Shotton et al., CVPR, 2008
[26]: J. Shotton et al., ECCV, 2006
[1]: D. Batra et al., CVPR, 2008
[25]: L. Yang et al., CVPR, 2007
![Page 35: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/35.jpg)
35
Results on The VOC-2008
![Page 36: Associative Hierarchical CRFs for Object Class Image Segmentation L’ubor Ladick’y and Chris Russell Oxford Brookes University Pushmeet Kohli Microsoft.](https://reader036.fdocuments.us/reader036/viewer/2022062409/5697bf721a28abf838c7ec6f/html5/thumbnails/36.jpg)
36
Results on The VOC-2008