Jifeng Dai

2011/09/27

Introduction

Structural SVM

Kernel Design

Segmentation and parameter learning

Object Feature Descriptors

Experimental results

Conclusions and Future Work

CVPR 2011 Oral

Things to do:

Contributions:

1) Propose a kernelized structural support vector machine approach to learn discriminatively the mapping from image to a segmentation mask.

2) Combine high level object similarity information with multiple low level segmentation cues into a novel kernel.

3) Traditional segmentation regularizations are preserved in the framework and explicitly enforced during the learning process. This way smoothness of the solution does not need to be “re-learned” from training examples.

Complex output

The dog chased the catxS VPNP

Det NV

NP

Det N

y2

S VPVP

Det NV

NP

V N

y1

S

NPVP

Det NV

NP

Det N

yk

…

Training Examples:

Hypothesis Space:

The dog chased the catx

S VPNP

Det NV

NP

Det N

y1

S VPVP

Det NV

NP

V N

y2

S

NPVP

Det NV

NP

Det N

y58

S VPNP

Det NV

NP

Det N

y12

S VPNP

Det NV

NP

Det N

y34

S VPNP

Det NV

NP

Det N

y4

Training: Find that solve

Problems• How to predict efficiently?• How to learn efficiently?• Manageable number of parameters?

The idea behind Structured SVM is to discriminatively learn a scoring function over input/output pairs (i.e. over image/mask pairs).

Loss function:

Two important choices:1) Restrict the search to Ys, subset of Y

composed by smooth segmentation masks.

Two important choices:1) Restrict the search to Ys, subset of Y

composed by smooth segmentation masks.

2) using kernel functions so that we could work in the dual formulation.

HOG…

Object Similarity KernelMask Similarity Kernel

Mask Similarity Kernel

1) Shape Kernel

2) Local Color Model Kernel

3) Global Color Model Kernel

Graph cuts

Mask smooth term

In which

So (6) and (7) take the form:

Graph cuts!!!

Parameters are optimized on a validation set

HOG grid or detector response feature

Datasets:

1)the Dresses dataset (600 images)2)the Weizmann horses dataset (328 images)3)the Oxford 17 category flower dataset (849

images)

How to measure performance?

Comparison with previous works:

Comparison with previous works:

Comparison with previous works: Oxford Flower Dataset

Previous work:

Examples:

Examples:

Examples:

Contributions:

1) Propose a kernelized structural support vector machine approach to learn discriminatively the mapping from image to a segmentation mask.2) Combine high level object similarity information with multiple low level segmentation cues into a novel kernel.3) Traditional segmentation regularizations are preserved in the framework and explicitly enforced during the learning process. This way smoothness of the solution does not need to be “re-learned” from training examples.

Future Work:

1)Model the boundary curves (driven by low-level cues).

2) Instead of relying on a single global object similarity kernel, dividing the kernel into a parts-based representation.

3) Establish a theoretical connection between the complexity of the top-down models the algorithm can learn and the number of segmentations needed in the training set.