Andrew Rosenberg- Lecture 21: Spectral Clustering

8/3/2019 Andrew Rosenberg- Lecture 21: Spectral Clustering

1/43

Lecture21:SpectralClustering

April22,2010


2/43

LastTime

GMMModelAdapta=onMAP(MaximumAPosteriori)MLLR(MaximumLikelihoodLinearRegression)

UMB-MAPforspeakerrecogni=on


3/43

Today

GraphBasedClusteringMinimumCut


4/43

Par==onalClustering

Howdowepar==onaspacetomakethebestclusters?

Proximitytoaclustercentroid.


5/43

DifficultClusterings

Butsomeclusteringsdontlendthemselvestoacentroidbaseddefini=onofacluster.

Spectralclusteringallowsustoaddressthesesortsofclusters.


6/43

DifficultClusterings

Thesekindsofclustersaredefinedbypointsthatarecloseanymemberinthecluster,

ratherthantheaveragememberofthecluster.


7/43

GraphRepresenta=on

Wecanrepresenttherela=onshipsbetweendatapointsinagraph.


8/43

GraphRepresenta=on

Wecanrepresenttherela=onshipsbetweendatapointsinagraph.

Weighttheedgesbythesimilaritybetweenpoints


9/43

Represen=ngdatainagraph

Whatisthebestwaytocalculatesimilaritybetweentwodatapoints?

Distancebased:d(xi, xj) = expxi xj

2


10/43

Graphs

NodesandEdges Edgescanbedirectedorundirected Edgescanhaveweightsassociatedwiththem

Heretheweightscorrespondtopairwiseaffinitywij = d(xi, xj) = exp

xi xj

2

wijxi

xj


11/43

Graphs

Degree

Volumeofaset

D(xi) =

jV

wij

V ol(C) =

iC

D(xi)


12/43

GraphCuts

Thecutbetweentwosubgraphsiscalculatedasfollows

Cut(C1, C2) =

iC1

jC2

wij


13/43

GraphExamples-Distance

Height Weight

20 5

8 6

9 4

4 4

4 51

5D

E

C

5.1

2.2

B

4.5

4.1

A

12

11


14/43

GraphExamples-Similarity

Height Weight

20 5

8 6

9 4

4 4

4 51

.2D

E

C

.19

.45

B

.22

.24

A

.08

.09


15/43

Intui=on

Theminimumcutofagraphiden=fiesanop=malpar==oningofthedata.

SpectralClusteringRecursivelypar==onthedataset

Iden=fytheminimumcut Removeedges Repeatun=lkclustersareiden=fied


16/43

GraphCuts

Minimum(bipar==onal)cutminCut(C1, C2) =

iC1

jC2

wij


17/43

GraphCuts

Minimum(bipar==onal)cutminCut(C1, C2) =

iC1

jC2

wij


18/43

GraphCuts

Minimal(bipar==onal)normalizedcut.

Unnormalizedcutsarearactedtooutliers.

minCut(C1, C2)

V ol(C1)+

Cut(C1, C2)

V ol(C2)= min

1

V ol(C1)+

1

V ol(C2)Cut(C1, C2)


19/43

Graphdefini=ons

-neighborhoodgraphIden=fyathresholdvalue,,andincludeedgesif

theaffinitybetweentwopointsisgreaterthan.

k-nearestneighborsInsertedgesbetweenanodeanditsk-nearest

neighbors.

Eachnodewillbeconnectedto(atleast)knodes. Fullyconnected

Insertanedgebetweeneverypairofnodes.


20/43

Intui=on

Theminimumcutofagraphiden=fiesanop=malpar==oningofthedata.

SpectralClusteringRecursivelypar==onthedataset

Iden=fytheminimumcut Removeedges Repeatun=lkclustersareiden=fied


21/43

SpectralClusteringExample

MinimumCutHeight Weight

20 5

8 6

9 4

4 4

4 51

.2D

E

C

.19

.45

B

.22

.24

A

.08

.09

Cut(BCDE,A) = 0.17


22/43


NormalizedMinimumCutHeight Weight

20 5

8 69 4

4 4

4 51

.2D

E

C

.19

.45

B

.22

.24

A

.08

.09

NormCut(BCDE,A) = 1.067

NormCut(C1, C2) =Cut(C1, C2)

V ol(C1)+

Cut(C1, C2)

V ol(C2)


23/43


NormalizedMinimumCutHeight Weight

20 5

8 69 4

4 4

4 51

.2D

E

C

.19

.45

B

.22

.24

A

.08

.09

NormCut(BCDE,A) = 1.067

NormCut(ABC,DE) = 1.038

NormCut(C1, C2) =Cut(C1, C2)

V ol(C1)+

Cut(C1, C2)

V ol(C2)


24/43

Problem

Iden=fyingaminimumcutisNP-hard. Thereareefficientapproxima=onsusinglinear

algebra. BasedontheLaplacianMatrix,orgraphLaplacian


25/43

SpectralClustering

ConstructanaffinitymatrixA B C D

A .4 .2 .2 0

B .2 .5 .3 0

C .2 .3 .6 .1

D 0 0 .1 .1

.1

.3

.2.2

A

B C

DW =


26/43

SpectralClustering

ConstructthegraphLaplacian

Iden=fyeigenvectorsoftheaffinitymatrix

D = diag(D0, D1, . . . , Dn)

L = DW

Lv = v


27/43

SpectralClustering

K-Meansoneigenvectortransforma=onofthedata.

Projectbacktotheini=aldatarepresenta=on.

k-eigenvectors

n-datapoints

EachRowrepresentsa

datapointintheeigenvector

space


28/43

Overview:whatarewedoing?

Definetheaffinitymatrix Iden=fyeigenvaluesandeigenvectors. K-meansoftransformeddata Projectbacktooriginalspace


29/43

Whydoesthiswork?

IdealCase

Whatareweop=mizing?Whydotheeigenvectorsofthelaplacianincludeclusteriden=fica=oninforma=on

1 0

1 0

1 0

0 1

0 1

0 1

1 1 1 0 0 0

1 1 1 0 0 0

1 1 1 0 0 0

0 0 0 1 1 1

0 0 0 1 1 1

0 0 0 1 1 1

Lv = v


30/43

Whydoesthiswork?

Howdoesthiseigenvectordecomposi=onaddressthis?

ifweletfbeeigenvectorsofL,thentheeigenvaluesaretheclusterobjec=vefunc=ons

f(xj) = fj clusterassignmentL = D W

fT

Lf = fT

DffT

Wf

=i

dif2

i

ij

fifjwij

=1

2

i

j

wij

f2i 2ij

fifjwij +ji

wij

f2j

=1

2

ij

wij(fi fj)2

Clusterobjec=vefunc=onnormalizedcut!


31/43

NormalizedGraphCutsview

Minimal(bipar==onal)normalizedcut.

Eigenvaluesofthelaplacianareapproximatesolu=onstomincutproblem.

minCut(C1, C2)

V ol(C1)

+Cut(C1, C2)

V ol(C2)

= min1

V ol(C1)

+1

V ol(C2)Cut(C1, C2)

NCut(A,B) =yT(D W)y

yTDy

minyyT(D W)y subject to yTDy = 1

(DW)y = Dy


32/43

TheLaplacianMatrix

LD-W Posi=vesemi-definite Thelowesteigenvalueis0,eigenvectoris Thesecondlowestcontainsthesolu=on

Thecorrespondingeigenvectorcontainstheclusterindicatorforeachdatapoint

xTLx 0

1

2 =Cut(A,B)

|A|+Cut(A,B)

|B|


33/43

Usingeigenvectorstopar==on

Eacheigenvectorpar==onsthedatasetintotwoclusters.

Theentryinthesecondeigenvectordeterminesthefirstcut.

Subsequenteigenvectorscanbeusedtofurtherpar==onintomoresets.


34/43

Example

Denseclusterswithsomesparseconnec=ons


35/43

3classExample

Affinitymatrix eigenvectors rownormaliza=on output


36/43

Example[Ngetal.2001]


37/43

k-meansvs.SpectralClustering

K-means SpectralClustering


38/43

Randomwalkviewofclustering

Inarandomwalk,youstartatanode,andmovetoanothernodewithsomeprobability.

Theintui=onisthatiftwonodesareinthesamecluster,youarandomlywalkislikelyto

reachboth

points.


39/43

Randomwalkviewofclustering

Transi=onmatrix: Thetransi=onprobabilityisrelatedtotheweightofgiventransi=onandtheinverse

degreeofthe

currentnode.

Lrw= ID

1W = D

1L


40/43

Usingminimumcutforsemi

supervisedclassifica=on? Constructagraphrepresenta=onofunseendata. Insertimaginarynodessandtconnectedtolabeledpoints

withinfinitesimilarity.

Treatthemincutasamaximumflowproblemfromstot

s

t


41/43

KernelMethod

Theweightbetweentwonodesisdefinedasafunc=onoftwodatapoints.

Wheneverwehavethis,wecanuseanyvalidKernel.

wij = d(xi, xj) = expxi xj

2

wij = K(xi, xj)


42/43

Today

Graphrepresenta=onsofdatasetsforclustering

SpectralClustering


43/43

NextTime

Evalua=on.Classifica=onClustering

Andrew Rosenberg- Lecture 21: Spectral Clustering

Documents

Transcript of Andrew Rosenberg- Lecture 21: Spectral Clustering