MachineLearning

Slides: James Hays, Isabelle Guyon, Erik Sudderth, Mark Johnson, Derek Hoiem Photo: CMU Machine Learning

Department protests G20

Clustering:grouptogethersimilarpointsandrepresentthemwithasingletoken

KeyChallenges:1)Whatmakestwopoints/images/patchessimilar?2)HowdowecomputeanoverallgroupingfrompairwisesimilariCes?

Slide: Derek Hoiem

Howdowecluster?

•  K-means–  IteraCvelyre-assignpointstothenearestclustercenter

•  AgglomeraCveclustering–  StartwitheachpointasitsownclusteranditeraCvelymergetheclosestclusters

•  Mean-shiHclustering–  EsCmatemodesofpdf

•  Spectralclustering–  Splitthenodesinagraphbasedonassignedlinkswithsimilarityweights

ClusteringforSummarizaConGoal:clustertominimizevarianceindatagivenclusters– PreserveinformaCon

( )∑∑ −=N

j

K

ijiN ij

21

,

** argmin, xcδcδc

δ

Whether xj is assigned to ci

Cluster center Data

Slide: Derek Hoiem

K-meansalgorithm

Illustration: http://en.wikipedia.org/wiki/K-means_clustering

1. Randomly select K centers

2. Assign each point to nearest center

3. Compute new center (mean) for each cluster

K-meansalgorithm

Illustration: http://en.wikipedia.org/wiki/K-means_clustering

1. Randomly select K centers

2. Assign each point to nearest center

3. Compute new center (mean) for each cluster

Back to 2

BuildingVisualDicConaries1.  Samplepatchesfrom

adatabase–  E.g.,128dimensional

SIFTvectors

2.  Clusterthepatches–  Clustercentersare

thedicConary

3.  Assignacodeword(number)toeachnewpatch,accordingtothenearestcluster

Examplesoflearnedcodewords

Sivic et al. ICCV 2005 http://www.robots.ox.ac.uk/~vgg/publications/papers/sivic05b.pdf

Most likely codewords for 4 learned “topics” EM with multinomial (problem 3) to get topics

AgglomeraCveclustering

AgglomeraCveclustering

AgglomeraCveclustering

AgglomeraCveclustering

AgglomeraCveclustering

AgglomeraCveclusteringHowtodefineclustersimilarity?-  Averagedistancebetweenpoints,

maximumdistance,minimumdistance-  Distancebetweenmeansormedoids

Howmanyclusters?-  Clusteringcreatesadendrogram(atree)-  Thresholdbasedonmaxnumberofclusters

orbasedondistancebetweenmergesdi

stan

ce

AgglomeraCveclusteringdemohZp://home.dei.polimi.it/maZeucc/Clustering/tutorial_html/AppletH.html

Conclusions:AgglomeraCveClusteringGood•  Simpletoimplement,widespreadapplicaCon•  ClustershaveadapCveshapes•  ProvidesahierarchyofclustersBad•  Mayhaveimbalancedclusters•  SCllhavetochoosenumberofclustersorthreshold

•  Needtousean“ultrametric”togetameaningfulhierarchy

•  VersaCletechniqueforclustering-basedsegmentaCon

D. Comaniciu and P. Meer, Mean Shift: A Robust Approach toward Feature Space Analysis, PAMI 2002.

MeanshiHsegmentaCon

MeanshiHalgorithm•  Trytofindmodesofthisnon-parametric

density

KerneldensityesCmaCon

Kernel density estimation function

Gaussian kernel

Region of interest

Center of mass

Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel

MeanshiH

Region of interest

Center of mass

Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel

MeanshiH

Region of interest

Center of mass

Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel

MeanshiH

Region of interest

Center of mass

Mean Shift vector

MeanshiH

Slide by Y. Ukrainitz & B. Sarel

Region of interest

Center of mass

Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel

MeanshiH

Region of interest

Center of mass

Mean Shift vector

Slide by Y. Ukrainitz & B. Sarel

MeanshiH

Region of interest

Center of mass

Slide by Y. Ukrainitz & B. Sarel

MeanshiH

Simple Mean Shift procedure: •  Compute mean shift vector

• Translate the Kernel window by m(x)

2

1

2

1

( )

ni

ii

ni

i

gh

gh

=

=

⎡ ⎤⎛ ⎞⎢ ⎥⎜ ⎟

⎜ ⎟⎢ ⎥⎝ ⎠= −⎢ ⎥⎛ ⎞⎢ ⎥⎜ ⎟⎢ ⎥⎜ ⎟⎝ ⎠⎣ ⎦

∑

∑

x - xx

m x xx - x

g( ) ( )kʹ= −x x

CompuCngtheMeanShiH

Slide by Y. Ukrainitz & B. Sarel

•  AZracConbasin:theregionforwhichalltrajectoriesleadtothesamemode

•  Cluster:alldatapointsintheaZracConbasinofamode

Slide by Y. Ukrainitz & B. Sarel

AZracConbasin

AZracConbasin

MeanshiHclustering•  ThemeanshiHalgorithmseeksmodesofthe

givensetofpoints1.  Choosekernelandbandwidth2.  Foreachpoint:

a)  Centerawindowonthatpointb)  Computethemeanofthedatainthesearchwindowc)  CenterthesearchwindowatthenewmeanlocaCond)  Repeat(b,c)unClconvergence

3.  Assignpointsthatleadtonearbymodestothesamecluster

•  Computefeaturesforeachpixel(color,gradients,texture,etc)•  SetkernelsizeforfeaturesKfandposiConKs•  IniCalizewindowsatindividualpixellocaCons•  PerformmeanshiHforeachwindowunClconvergence•  MergewindowsthatarewithinwidthofKfandKs

SegmentaConbyMeanShiH

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

MeanshiHsegmentaConresults

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

Mean-shiH:otherissues•  Speedups

–  BinnedesCmaCon–  Fastsearchofneighbors– UpdateeachwindowineachiteraCon(fasterconvergence)

•  Othertricks– UsekNNtodeterminewindowsizesadapCvely

•  LotsoftheoreCcalsupportD.ComaniciuandP.Meer,MeanShiH:ARobustApproachtowardFeatureSpaceAnalysis,PAMI2002.

MeanshiHprosandcons

•  Pros–  Goodgeneral-pracCcesegmentaCon–  Flexibleinnumberandshapeofregions–  Robusttooutliers

•  Cons–  Havetochoosekernelsizeinadvance–  Notsuitableforhigh-dimensionalfeatures

•  Whentouseit–  Oversegmentatoin– MulCplesegmentaCons–  Tracking,clustering,filteringapplicaCons

Spectralclustering Grouppointsbasedonlinksinagraph

A B

Cutsinagraph

A B

Normalized Cut •  a cut penalizes large segments •  fix by normalizing for size of segments

•  volume(A) = sum of costs of all edges that touch A

Source: Seitz

NormalizedcutsforsegmentaCon

VisualPageRank•  Determiningimportancebyrandomwalk

– What’stheprobabilitythatyouwillrandomlywalktoagivennode?

•  Createadjacencymatrixbasedonvisualsimilarity•  EdgeweightsdetermineprobabilityoftransiCon

Jing Baluja 2008

Whichalgorithmtouse?•  QuanCzaCon/SummarizaCon:K-means

– Aimstopreservevarianceoforiginaldata– Caneasilyassignnewpointtoacluster

Quantization for computing histograms

Summary of 20,000 photos of Rome using “greedy k-means”

http://grail.cs.washington.edu/projects/canonview/

Whichalgorithmtouse?•  ImagesegmentaCon:agglomeraCveclustering

– Moreflexiblewithdistancemeasures(e.g.,canbebasedonboundarypredicCon)

– AdaptsbeZertospecificdata– Hierarchycanbeuseful

http://www.cs.berkeley.edu/~arbelaez/UCM.html

Thingstoremember•  K-meansusefulforsummarizaCon,buildingdicConariesofpatches,generalclustering

•  AgglomeraCveclusteringusefulforsegmentaCon,generalclustering

•  Spectralclusteringusefulfordeterminingrelevance,summarizaCon,segmentaCon

ClusteringKeyalgorithm•  K-means