CSSE463: Image Recognition CSSE463: Image Recognition Day 21Day 21 Sunset detector due Wednesday night. Sunset detector due Wednesday night.

Foundations of Image Recognition completedFoundations of Image Recognition completed This week: This week:

K-means: a method of image segmentationK-means: a method of image segmentation Template matching: a simple method for object detectionTemplate matching: a simple method for object detection

Questions?Questions?

An image to segment…

SegmentationSegmentation The process of breaking an The process of breaking an

image into regions.image into regions. Two types: Two types:

General-purposeGeneral-purpose ““One size fits all”One size fits all” Very difficult…Very difficult…

SpecializedSpecialized Intended for a specific domain Intended for a specific domain

(say fruit-, circle- or skin-(say fruit-, circle- or skin-finding)finding)

Can be successfulCan be successful One to right is created using the One to right is created using the

mean-shift algorithm mean-shift algorithm D. Comaniciu, P. Meer: D. Comaniciu, P. Meer:

Mean shift: A robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Machine IntellIEEE Trans. Pattern Anal. Machine Intell, , 2424, , 603-619, 2002.603-619, 2002.

EDISON code downloadable at EDISON code downloadable at

http://www.caip.rutgers.edu/riul/http://www.caip.rutgers.edu/riul/research/robust.htmlresearch/robust.html

What properties can we use to What properties can we use to segment?segment?

Regions homogeneous wrt. color, texture, Regions homogeneous wrt. color, texture, etc.etc.

Simple interiorsSimple interiorsAdjacent regions different (else merge)Adjacent regions different (else merge)Smooth boundariesSmooth boundaries

ApproachesApproaches

1.1. ModelsModels Uses an expected shape, color, etc. (fruit- and Uses an expected shape, color, etc. (fruit- and

circle-finders)circle-finders) Can use probabilitiesCan use probabilities

2.2. ClusteringClustering An An unsupervised unsupervised machine learning techniquemachine learning technique

No class labels used in learning!No class labels used in learning!

Groups pixels “close” to each other by some metric.Groups pixels “close” to each other by some metric. Color distance, texture, intensity, spatial location, etc.Color distance, texture, intensity, spatial location, etc.

Regions are then found using connected Regions are then found using connected componentscomponents

Q1

K-means clusteringK-means clustering Goal:Goal: given K given K

clusters, assign each clusters, assign each pixel to one of the pixel to one of the clusters such that the clusters such that the total total distance from distance from each pixel to the each pixel to the center of its cluster is center of its cluster is minimized.minimized.

We control C, the assignment We control C, the assignment of pixels to each cluster. (We of pixels to each cluster. (We will actually do this by will actually do this by specifying the location of their specifying the location of their means)means)

D= total distanceD= total distance K = # of clustersK = # of clusters x are pixelsx are pixels CCkk is the set of pixels in is the set of pixels in

cluster cluster kk mmkk is the center of cluster is the center of cluster kk ||.|| is a distance||.|| is a distance

Q2

K

k Cxki

Cki

mxD1

2min

K-means clusteringK-means clusteringProblems: Problems:

What’s K?What’s K? How do we know How do we know

which pixel belongs to which pixel belongs to each cluster?each cluster?

K-means is an K-means is an answer to the second answer to the second question.question.

K = # of clustersK = # of clusters x are pixelsx are pixels CCkk is the set of pixels in is the set of pixels in

cluster cluster kk mmkk is the center of cluster is the center of cluster kk ||.|| is a distance||.|| is a distance

K

k Cxki

Cki

mxD1

2min

K-means clusteringK-means clustering

Iterative process to group into Iterative process to group into kk clusters. clusters. Algorithm Algorithm (Sonka, p 403; Forsyth&Ponce, p. 315; Shapiro, p. 282)(Sonka, p 403; Forsyth&Ponce, p. 315; Shapiro, p. 282)

Initialize k cluster meansInitialize k cluster means Repeat until convergence:Repeat until convergence:

For each pixel, find the closest mean and assign it For each pixel, find the closest mean and assign it to that cluster to that cluster

Re-compute the mean of all pixels assigned to the Re-compute the mean of all pixels assigned to the clustercluster

Label each pixel with its current clusterLabel each pixel with its current cluster Example on board using 2D spatial distanceExample on board using 2D spatial distance Could you implement? Could you implement?

Q3-5

K-means clusteringK-means clustering We are trying to find out where the clusters are and We are trying to find out where the clusters are and

which points are assigned to each cluster. We which points are assigned to each cluster. We iteratively solve half the problem. Notice the overall iteratively solve half the problem. Notice the overall structure:structure:

Repeat until convergence:Repeat until convergence:

Assume you know where the cluster centers are. Assume you know where the cluster centers are. For each For each pixel, find the closest mean and assign it to that cluster pixel, find the closest mean and assign it to that cluster

Assume you know which points belong to each cluster.Assume you know which points belong to each cluster. Recompute the mean of all pixels assigned to the clusterRecompute the mean of all pixels assigned to the cluster

Label each pixel with its current clusterLabel each pixel with its current cluster

K-means clusteringK-means clustering Pros:Pros:

Easy to implementEasy to implement Finds local optimumFinds local optimum Global optimization (over all partitions) is infeasibleGlobal optimization (over all partitions) is infeasible

Cons:Cons: The number of clusters, The number of clusters, kk, must be known in advance, must be known in advance Some clusters might have 0 pointsSome clusters might have 0 points Local optimum is only an approximationLocal optimum is only an approximation

Ideas:Ideas: Can re-run with several initializationsCan re-run with several initializations Can choose Can choose kk based on observation or statistical means based on observation or statistical means

Q6

K-means clusteringK-means clustering

K = # of clustersK = # of clusters x are pixelsx are pixels CCkk is the set of pixels in is the set of pixels in

cluster cluster kk mmkk is the center of cluster is the center of cluster kk ||.|| is a distance: could be 2D ||.|| is a distance: could be 2D

distance in image or distance in image or 3D 3D Euclidean distance between Euclidean distance between colorscolors (or combination of both) (or combination of both)

(On Lab: will produce disconnected regions)

Q7-8

K

k Cxki

Cki

mxD1

2min

K-means resultsK-means results

K=3Original (120x160)

K=5 K=7