C. Ventura X. Giró-i-Nieto V. Vilaplana F. Marqués K. McGuinness N. O’Connor

Improving Spatial Codification in Semantic Segmentation

Outline

● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions

Introduction● Object segmentation vs Class segmentation

Image Object segmentation

Class segmentation

3

Outline

● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions

Related Work● The Visual Extent of

an Object [1]

5[1] Uijlings et al, The VIsual Extent of an Object. IJCV’12

1st Contribution● Using a Figure-Border-Ground spatial pooling with object

candidates

6

Figure-Ground spatial pooling

Figure-Border-Ground spatial pooling

Related Work● Beyond bags of features:

Spatial pyramid matching for recognizing natural scene categories [1]

7[1] Lazebnik et al, Beyond bags of features: Spatial pyramid matching for recognizing natural scenes. CVPR’06

Related Work● Variations of SPM

○ Non-arbitrary division■ Object-centric pooling [1]■ Object confidence map

partition [2]○ SPM over bounding boxes [3]

[4]

[1] Russakovky et al, Object-centric spatial pooling for image classification. ECCV’12[2] Chen et al, Hierarchical Matching with Side Information for Image Classification. CVPR’12[3] Arbeláez et at, Semantic segmentation using regions and parts. CVPR’12[4] Gu et al, Multi-component models for object detection. ECCV’12 8

2nd Contribution● Applying a contour-based spatial pyramid (SP)

○ Crown-based SP○ Cartesian-based SP

9

Crown-basedspatial pyramid

Cartesian-basedspatial pyramid

Outline

● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions

Architecture● Architecture proposed and released in [1]

[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12

Train Test

DataBaseObject

CandidatesFeature

Extraction

Test

Model

Prediction

Evaluation

AAC

Ground Truth

Train

CPMC SIFT-based features (O2P)

11

Outline

● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions

Experiments● Experiments with ideal object

candidates○ Train set: train11○ Test set: val11

F[1] F-B F-G[1] F-B-G

eSIFT 63.9 66.2 66.4 68.6

eMSIFT 64.8 68.9 67.6 70.8

[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 13

Experiments● Experiments with ideal object

candidates○ Train set: train11○ Test set: val11

F F-B F-B-G

non SP 64.8 [1] 68.9 70.8

crown-based SP 68.7 71.1 71.7

Cartesian-based SP 67.7 71.6 72.7

Figure SP(Figure) Border Ground AAC

eSIFT+eMSIFT+eLBP eSIFT 72.98 [1]

eSIFT+eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 73.84

eSIFT+eMSIFT+eLBP eMSIFT eMSIFT+eSIFT eMSIFT+eSIFT 75.86

[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’1214

Experiments● Experiments with CPMC object candidates

○ Train set: train11○ Test set: val11

[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12

Figure SP(Figure) Border Ground AAC

eSIFT eSIFT 28.6 [1]

eSIFT eSIFT eSIFT 34.8

eSIFT+eMSIFT+eLBP eSIFT 37.2 [1]

eSIFT eSIFT eSIFT eSIFT 37.4

eSIFT+eMSIFT+eLBP eSIFT eSIFT eSIFT 39.6

15

Experiments● Experiments with CPMC object

candidates in comp5 challenge○ Train set: trainval11 /

trainval12○ Test set: test11 / test12

F-G [1] F-B-G SP(F)-B-G

PASCAL VOC11

38.8 43.8 40.3

PASCAL VOC12

39.9 42.2 40.8

[1] Carreira et al, Semantic segmentation with second-order pooling. ECCV’1216

Experiments● Experiments with MCG object

candidates [1]○ Train set: train11○ Test set: val11

F-G[2] F-B-G SP(F)-B-G

CPMC 37.2 38.9 39.6

MCG 30.9 34.1 36.1

[1] Arbeláez et al, Multiscale Combinatorial Grouping. CVPR’14[2] Carreira et al, Semantic segmentation with second-order pooling. ECCV’12 17

Experiments● Qualitative results F-B-G spatial pooling with CPMC

18

F-G F-B-G F-G F-B-G

aeroplane

bicycle bicycle

birdcat

motorbike boat

bottle

bus bus

motorbike

car

chaircat

chair chair

horse

cow

bird

Experiments● Qualitative results F-B-G spatial pooling with CPMC

19

chair

dining table

cow dog

person

horse

person motorbike

motorbikemotorbike

person

plotted plant bottle

sheep

sofa dog

bus

train train

tvmonitor

F-G F-B-G F-G F-B-G

Outline

● Introduction● Related Work and Contributions● Architecture● Experiments● Conclusions

Conclusions● 2 proposals beyond the classic Figure-Ground pooling

○ Figure-Border-Ground spatial pooling■ Extended to realistic scenario with CPMC object

candidates○ A novel contour-based spatial pyramid has been introduced

■ Cartesian-based spatial pyramid■ Crown-based spatial pyramid

● Validation of both proposals also for MCG object candidates

21

Related Work● The Visual

Extent of an Object (Uijlings et al, IJCV’12)

23