Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and...
-
Upload
anne-hardy -
Category
Documents
-
view
221 -
download
0
Transcript of Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and...
![Page 1: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/1.jpg)
Segmentation using eigenvectors
Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000
“Segmentation using eigenvectors: a unifying view”. Yair Weiss, ICCV 1999.
Presenter: Carlos Vallespi
![Page 2: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/2.jpg)
Image Segmentation
![Page 3: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/3.jpg)
Image segmentation
How do you pick the right segmentation?
•Bottom up segmentation: - Tokens belong together because they are locally coherent. •Top down segmentation: - Tokens grouped because they lie on the same object.
•Bottom up segmentation: - Tokens belong together because they are locally coherent. •Top down segmentation: - Tokens grouped because they lie on the same object.
![Page 4: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/4.jpg)
“Correct” segmentation
There may not be a single correct answer.
Partitioning is inherently hierarchical.One approach we will use in this
presentation: “Use the low-level coherence of brightness,
color, texture or motion attributes to come up with partitions”
![Page 5: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/5.jpg)
Outline
1. Introduction
2. Graph terminology and representation.
3. “Min cuts” and “Normalized cuts”.
4. Other segmentation methods using eigenvectors.
5. Conclusions.
![Page 6: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/6.jpg)
Outline
1.1. IntroductionIntroduction
2. Graph terminology and representation.
3. “Min cuts” and “Normalized cuts”.
4. Other segmentation methods using eigenvectors.
5. Conclusions.
![Page 7: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/7.jpg)
Graph-based Image Segmentation
Image (I)
Graph Affinities(W)
IntensityColorEdgesTexture
Slide from Timothee Cour (http://www.seas.upenn.edu/~timothee)
![Page 8: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/8.jpg)
Graph-based Image Segmentation
Image (I)
)(
1
)(
1),(),(
BvolAvolBAcutBANcut
Slide from Timothee Cour (http://www.seas.upenn.edu/~timothee)
Graph Affinities(W)
IntensityColorEdgesTexture
![Page 9: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/9.jpg)
Graph-based Image Segmentation
Image (I)
)(
1
)(
1),(),(
BvolAvolBAcutBANcut
Eigenvector X(W)
Aiif
AiifiX
DXXWD
A 0
1)(
)(
Slide from Timothee Cour (http://www.seas.upenn.edu/~timothee)
Graph Affinities(W)
IntensityColorEdgesTexture
![Page 10: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/10.jpg)
Graph-based Image Segmentation
Image (I)
)(
1
)(
1),(),(
BvolAvolBAcutBANcut
Eigenvector X(W)
Discretization
Slide from Timothee Cour (http://www.seas.upenn.edu/~timothee)
Aiif
AiifiX
DXXWD
A 0
1)(
)(
Graph Affinities(W)
IntensityColorEdgesTexture
![Page 11: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/11.jpg)
Outline
1. Introduction
2.2. Graph terminology and representation.Graph terminology and representation.
3. “Min cuts” and “Normalized cuts”.
4. Other segmentation methods using eigenvectors.
5. Conclusions.
![Page 12: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/12.jpg)
Graph-based Image Segmentation
V: graph nodesE: edges connection nodes
G = {V,E}
PixelsPixel similarity
Slides from Jianbo Shi
![Page 13: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/13.jpg)
Graph terminology
Similarity matrix:
Slides from Jianbo Shi
2
2
2)()(
,X
ji XX
ji ew
jiwW ,
![Page 14: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/14.jpg)
Affinity matrix
Similarity of image pixels to selected pixelBrighter means more similar
Reshape
N*M pixels
N*M pixels
M pixels
N pixels
WarningWarningthe size of W is quadratic
with the numberof parameters!
WarningWarningthe size of W is quadratic
with the numberof parameters!
![Page 15: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/15.jpg)
Graph terminology
Degree of node:
Slides from Jianbo Shi
j
jii wd ,
……
![Page 16: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/16.jpg)
Graph terminology
Volume of set:
Slides from Jianbo Shi
Ai
i VAdAvol ,)(
![Page 17: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/17.jpg)
Graph terminology
AjAi
jiwAAcut,
,),(
Slides from Jianbo Shi
Cuts in a graph:
![Page 18: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/18.jpg)
Representation
Partition matrix Partition matrix XX::
Pair-wise similarity matrix Pair-wise similarity matrix WW::
Degree matrix Degree matrix DD::
Laplacian matrix Laplacian matrix LL::
KXXX ,...,1
j jiwiiD ,),(
WDL
segments
pix
els
),(),( jiaffjiW
![Page 19: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/19.jpg)
Pixel similarity functions
Intensity
Texture
Distance
2
2
2)()(
),( I
ji II
ejiW
2
2
2)()(
),( X
ji XX
ejiW
2
2
2)()(
),( c
ji cc
ejiW
![Page 20: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/20.jpg)
Pixel similarity functions
Intensity
Texture
Distance
2
2
2)()(
),( I
ji II
ejiW
2
2
2)()(
),( X
ji XX
ejiW
2
2
2)()(
),( c
ji cc
ejiW
here c(x) is a vector of filter outputs. A natural thing to do is to square the outputs
of a range of different filters at different scales and orientations,
smooth the result, and rack these into a vector.
![Page 21: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/21.jpg)
Definitions
Methods that use the spectrum of the affinity matrix to cluster are known as spectral clusteringspectral clustering.
Normalized cuts, Average cuts, Average association make use of the eigenvectors of the affinity matrix.
Why these methods work?
![Page 22: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/22.jpg)
Spectral Clustering
Data Similarities
* Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University
![Page 23: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/23.jpg)
Eigenvectors and blocks
Block matrices have block eigenvectors:
Near-block matrices have near-block eigenvectors:
1 1 0 0
1 1 0 0
0 0 1 1
0 0 1 1
eigensolver
.71
.71
0
0
0
0
.71
.71
1= 2 2= 2 3= 0
4= 0
1 1 .2 0
1 1 0 -.2
.2 0 1 1
0 -.2 1 1
eigensolver
.71
.69
.14
0
0
-.14
.69
.71
1= 2.02 2= 2.02 3= -0.02
4= -0.02
* Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University
![Page 24: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/24.jpg)
Spectral Space
Can put items into blocks by eigenvectors:
Clusters clear regardless of row ordering:
1 1 .2 0
1 1 0 -.2
.2 0 1 1
0 -.2 1 1
.71
.69
.14
0
0
-.14
.69
.71
e1
e2
e1 e2
1 .2 1 0
.2 1 0 1
1 0 1 -.2
0 1 -.2 1
.71
.14
.69
0
0
.69
-.14
.71
e1
e2
e1 e2
* Slides from Dan Klein, Sep Kamvar, Chris Manning, Natural Language Group Stanford University
![Page 25: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/25.jpg)
Outline
1. Introduction
2. Graph terminology and representation.
3.3. ““Min cuts” and “Normalized cuts”.Min cuts” and “Normalized cuts”.
4. Other segmentation methods using eigenvectors.
5. Conclusions.
![Page 26: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/26.jpg)
How do we extract a good cluster?
Simplest ideaSimplest idea: we want a vector xx giving the association between each element and a cluster
We want elements within this cluster to, on the whole, have strong affinity with one anotherstrong affinity with one another
We could maximizemaximize But need the constraintconstraint This is an eigenvalue problemeigenvalue problem - choose the
eigenvector of W with largest eigenvalue.
WxxT
1xxT
![Page 27: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/27.jpg)
Criterion for partition:
Minimum cut
BvAu
BAvuwBAcut
,,
),(min),(min
First proposed by Wu and Leahy
A
B
Ideal Cut
Cuts with lesser weightthan the ideal cut
Problem! Problem! Weight of cut is directly proportional to the number of edges in the cut.
Problem! Problem! Weight of cut is directly proportional to the number of edges in the cut.
![Page 28: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/28.jpg)
Normalized Cut
)(
1
)(
1),(),(
BvolAvolBAcutBANcut
Normalized cut or balanced cut:
Finds better cut
![Page 29: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/29.jpg)
Normalized Cut
Volume of set (or association):
VtAutuwVAassocAvol
,),(),()(
A
B
![Page 30: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/30.jpg)
Normalized Cut
Volume of set (or association):
Define normalized cut: “a fraction of the total edge connections to all the nodes in the graph”:
),(
),(
),(
),(),(
VBassoc
BAcut
VAassoc
BAcutBANcut
VtAutuwVAassocAvol
,),(),()(
),(
),(
),(
),(),(
VBassoc
BBassoc
VAassoc
AAassocBANassoc
A
B
A
B
Define normalized association: “how tightly on average nodes within the cluster are connected to each other”
A
B
![Page 31: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/31.jpg)
01 DyTSubject to:
Observations(I)
Maximizing Nassoc is the same as minimizing Ncut, since they are related:
How to minimize Ncut? Transform Ncut equation to a matricial form. After simplifying:
),(2),( BANassocBANcut
Dyy
yWDyxNcut
T
T
yx
)(min)(min
Rayleigh quotient
j
jiWiiD ),(),( j
jiWiiD ),(),(
NP-Hard!NP-Hard!yy’s values are ’s values are
quantizedquantized
![Page 32: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/32.jpg)
Instead, relax into the continuous domain by solving generalized eigenvalue system:
Which gives: Note that so, the first eigenvector is
y0=1 with eigenvalue 0.
The second smallest eigenvector is the real valued solution to this problem!!
Observations(II)
DyyWD )(
01)( WD
maxy yT D W y subject to yT Dy 1 min
![Page 33: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/33.jpg)
Algorithm
1. Define a similarity function between 2 nodes. i.e.:
2. Compute affinity matrix (W) and degree matrix (D).3. Solve4. Use the eigenvector with the second smallest
eigenvalue to bipartition the graph.5. Decide if re-partition current partitions.
Note: since precision requirements are low, W is very sparse and only few eigenvectors are required, the eigenvectors can be extracted very fast using Lanczos algorithm.
DyyWD )(
2
2
2)()(
2
2
2)()(
,X
ji
I
ji XXFF
ji ew
![Page 34: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/34.jpg)
Discretization
Sometimes there is not a clear threshold to binarize since eigenvectors take on continuous values.
How to choose the splitting point? a) Pick a constant value (0, or 0.5).
b) Pick the median value as splitting point.
c) Look for the splitting point that has the minimum Ncut value:1. Choose n possible splitting points.
2. Compute Ncut value.
3. Pick minimum.
![Page 35: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/35.jpg)
Use k-eigenvectors
Recursive 2-way NcutNcut is slow. We can use more eigenvectors to re-partition the graph, however:
Not all eigenvectors are useful for partition (degree of smoothnessdegree of smoothness). Procedure: compute k-means k-means with a high kk. Then follow one of these
procedures:a) Merge segments that minimize kk-way NcutNcut criterion.
b) Use the k segments and find the partitions there using exhaustive search.
Compute Q (next slides).
1 1 .2 0
1 1 0 -.2
.2 0 1 1
0 -.2 1 1
.71
.69
.14
0
0
-.14
.69
.71
e1
e2
e1 e2
![Page 36: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/36.jpg)
Toy examples
Images from Matthew Brand (TR-2002-42)
![Page 37: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/37.jpg)
Example (I)
EigenvectorsEigenvectors
SegmentsSegments
![Page 38: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/38.jpg)
Example (II)
* Slide from Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Seg
men
tsS
egm
ents
OriginalOriginal
![Page 39: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/39.jpg)
Outline
1. Introduction
2. Graph terminology and representation.
3.3. ““Min cuts” and “Normalized cuts”.Min cuts” and “Normalized cuts”.
4.4. Other segmentation methods using Other segmentation methods using eigenvectors.eigenvectors.
5. Conclusions.
![Page 40: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/40.jpg)
Other methods
Average associationUse the eigenvector of W associated to the
biggest eigenvalue for partitioning.Tries to maximize:
Has a bias to find tight clusters. Useful for gaussian distributions.
B
BBassoc
A
AAassoc ),(),(
A
B
![Page 41: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/41.jpg)
Other methods
Average cutTries to minimize:
Very similar to normalized cuts.We cannot ensure that partitions will have a
a tight within-group similarity since this equation does not have the nice properties of the equation of normalized cuts.
B
BAcut
A
BAcut ),(),(
![Page 42: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/42.jpg)
Other methods
![Page 43: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/43.jpg)
Other methods
20 points are randomly distributed from 0.0 to 0.512 points are randomly distributed from 0.65 to 1.0
Normalized cutNormalized cut
Average cutAverage cut
Average associationAverage association
![Page 44: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/44.jpg)
Other methods
Scott and Longuet-Higgins (1990).Scott and Longuet-Higgins (1990). V contains the first eigenvectors of W. Normalize V by rows. Compute Q=VTV Values close to 1 belong to the same cluster.
Second evSecond evSecond evSecond evFirst evFirst evFirst evFirst ev QQQQWWWWDataDataDataData
![Page 45: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/45.jpg)
Other applications
Costeira and Kanade (1995).Costeira and Kanade (1995). Used to segment points in motion. Compute M=(XY). The affinity matrix W is compute as W=MTM. This trick
computes the affinity of every pair of points as a inner product. Compute Q=VTV Values close to 1 belong to the same cluster.
DataDataDataData MMMM QQQQ
![Page 46: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/46.jpg)
Other applications
Face clustering in meetings. Grab faces from video
in real time (use a face detector + face tracker).
Compare all faces using a distance metric (i.e. projection error into representative basis).
Use normalized cuts to find best clustering.
![Page 47: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/47.jpg)
Outline
1. Introduction
2. Graph terminology and representation.
3. “Min cuts” and “Normalized cuts”.
4. Other segmentation methods using eigenvectors.
5.5. Conclusions.Conclusions.
![Page 48: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/48.jpg)
Conclusions
Good news: Simple and powerful methods to segment images. Flexible and easy to apply to other clustering
problems.
Bad news: High memory requirements (use sparse matrices). Very dependant on the scale factor for a specific
problem.2
2
2)()(
),( X
ji XX
ejiW
![Page 49: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/49.jpg)
Thank you!Thank you!
The End!
![Page 50: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/50.jpg)
Examples
2
2
2)()(
,X
ji XX
ji ew
Spectral Clutering
Images from Matthew Brand (TR-2002-42)
![Page 51: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/51.jpg)
Spectral clustering
Makes use of the spectrum of the similarity matrix of the data to cluster the points.
Solve clustering for
affinity matrix
Solve clustering for
affinity matrix
w(i,j) distance node i to node j
![Page 52: Segmentation using eigenvectors Papers: “Normalized Cuts and Image Segmentation”. Jianbo Shi and Jitendra Malik, IEEE, 2000 “Segmentation using eigenvectors:](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649e795503460f94b798ee/html5/thumbnails/52.jpg)
Graph terminology
Similarity matrix: Degree of node: j
jii wd , jiwW ,
Volume of set: Graph cuts: