Mean Shift

Wednesday, 06/05/2020

Antonis Argyrose-mail: argyros@csd.uoc.gr

Mean shift clustering and segmentation

• An advanced and versatile technique for

clustering-based segmentation

D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature

Space Analysis, PAMI 2002.

What is Mean Shift ?

PDF in feature space

• Color space

• …

• Actually any feature space you can conceive

A tool for:

Finding modes in a set of data samples, manifesting an

underlying probability density function (PDF) in RN

Intuitive Description

Distribution of identical billiard balls

Region of

interest

Center of

mass

Mean Shift

vector

Objective : Find the densest region

Intuitive Description

Distribution of identical billiard balls

Region of

interest

Center of

mass

Mean Shift

vector

Objective : Find the densest region

Intuitive Description

Distribution of identical billiard balls

Region of

interest

Center of

mass

Mean Shift

vector

Objective : Find the densest region

Intuitive Description

Distribution of identical billiard balls

Region of

interest

Center of

mass

Mean Shift

vector

Objective : Find the densest region

Intuitive Description

Distribution of identical billiard balls

Region of

interest

Center of

mass

Mean Shift

vector

Objective : Find the densest region

Intuitive Description

Distribution of identical billiard balls

Region of

interest

Center of

mass

Mean Shift

vector

Objective : Find the densest region

Intuitive Description

Distribution of identical billiard balls

Region of

interest

Center of

mass

Objective : Find the densest region

What is Mean Shift ?

Non-parametric

Density Estimation

Non-parametric

Density GRADIENT Estimation

(Mean Shift)

Data

Discrete PDF Representation

PDF Analysis

A tool for:

Finding modes in a set of data samples, manifesting an

underlying probability density function (PDF) in RN

Non-Parametric Density Estimation

Assumption : The data points are sampled from an underlying PDF

Assumed Underlying PDF Real Data Samples

Data point density

implies PDF value !

Assumed Underlying PDF Real Data Samples

Non-Parametric Density Estimation

Assumed Underlying PDF Real Data Samples

Non-Parametric Density Estimation

Parametric Density Estimation

Assumption : The data points are sampled from an underlying PDF

Assumed Underlying PDF

2

2

( )

2

i

PDF( ) =

i

i

ic e

x-μ

x

Estimate

Real Data Samples

Real Modality Analysis

Tessellate the space

with windowsRun the procedure in parallel

Real Modality Analysis

The blue data points were traversed by the windows towards the mode

Clustering

Attraction basin : the region for which all trajectories lead to the same mode

Cluster : All data points in the attraction basin of a mode

Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer

Mean Shift Mode Detection

Updated Mean Shift Procedure:

• Find all modes using the Simple Mean Shift Procedure

• Prune modes by perturbing them (find saddle points and plateaus)

• Prune nearby – take highest mode in the window

What happens if we

reach a saddle point

?

Perturb the mode position

and check if we return back

ClusteringSynthetic Examples

Simple Modal Structures

Complex Modal Structures

ClusteringReal Example

L*u*v space representation

ClusteringReal Example

Initial window

centers

Modes found Modes after

pruning

Final clusters

Feature space:

L*u*v representation

SegmentationExample

…when feature space is only

gray levels…

SegmentationExample

SegmentationExample

SegmentationExample

SegmentationExample

SegmentationExample

SegmentationExample

Mean-Shift Object TrackingTarget Representation

Choose a

reference

target model

Quantized

Color Space

Choose a

feature space

Represent the

model by its

PDF in the

feature space

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 . . . m

color

Pro

bab

ilit

yKernel Based Object Tracking, by Comaniniu, Ramesh, Meer

,f p y q � �

Mean-Shift Object TrackingTarget Localization Algorithm

Start from the

position of the

model in the

current frame

q�

Search in the

model’s

neighborhood

in next frame

p y�

Find best

candidate by

maximizing a

similarity func.

Mean-Shift Object TrackingResults

Tracking Through Scale SpaceResults

Fixed-scale

Tracking through scale space

Mean shift

• Pros:– Does not assume shape on clusters

– One parameter choice (window size, aka “bandwidth”)

– Generic technique

– Find multiple modes

• Cons:– Selection of window size

– Does not scale well with dimension of feature space

Image Segmentation with Graph Cuts

q

Images as graphs

• Fully-connected graph

– node (vertex) for every pixel

– link between every pair of pixels, p,q

– affinity weight wpq for each link (edge)

• wpq measures similarity

• similarity is inversely proportional to difference (in color and

position…)

p

wpq

w

Source: Steve Seitz

Segmentation by Graph Cuts

• Break Graph into Segments

– Want to delete links that cross between segments

– Easiest to break links that have low similarity (low weight)

• similar pixels should be in the same segments

• dissimilar pixels should be in different segments

w

A B C

Source: Steve Seitz

q

p

wpq

Measuring affinity

• One possibility:

Small sigma:

group only

nearby points

Large sigma:

group distant

points

Measuring affinity

σ=.1 σ=.2 σ=1

σ=.2

Data points

Affinity

matrices

Slide credit: Kristen Grauman

Cuts in a graph: Min cut

• Link Cut

– set of links whose removal makes a graph

disconnected

Cost of a cut:

AB

Find minimum cut• gives you a segmentation

• fast algorithms exist for doing this

Source: Steve Seitz

BqAp

qpwBAcut,

,),(

Minimum cut

• Problem with minimum cut:

Weight of cut proportional to number of edges in the cut;

tends to produce small, isolated components.

J. Shi and J. Malik, Normalized Cuts and Image Segmentation, CVPR, 1997