Gaussian Shift (Mean Shift) Clustering and Variance...
Transcript of Gaussian Shift (Mean Shift) Clustering and Variance...
![Page 1: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/1.jpg)
Gaussian Shift (Mean Shift) Clustering and Variance Approximation
Jason Taillon
SUNYIT
31 March 2013
![Page 2: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/2.jpg)
Introduction
• Part 1
– Clustering
• Technique & Theory
• Part 2
– Variance Estimation
• Sub-grouping
• Displacement Analysis
![Page 3: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/3.jpg)
Overview & Background
• Mean Shift Clustering is a method of grouping data together, using gradient finding Algorithm
• Applications
– Genetics
• Clustering genes with similar expressions or motifs
– Machine Vision
• Grouping objects or people together
– Mathematics
• Finding groups or underlying distributions in data
![Page 4: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/4.jpg)
Problem?
• Given this data: • Are there clusters • How do we find them
![Page 5: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/5.jpg)
Algorithm 1
Given a set of points in a Cartesian plane
![Page 6: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/6.jpg)
Algorithm 2
Given a set of points in a Cartesian plane
1. Select a Starting Point
![Page 7: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/7.jpg)
Algorithm 2
Given a set of points in a Cartesian plane
1. Select a Starting Point 2. This point becomes a
local mean.
![Page 8: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/8.jpg)
Algorithm 3
Given a set of points in a Cartesian plane
1. Select a Starting Point 2. This point becomes a
local mean. 3. Cast a window with
radius R, about the local mean
![Page 9: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/9.jpg)
Algorithm 3
Given a set of points in a Cartesian plane 1. Select a Starting Point 2. This point becomes a local
mean. 3. Cast a window with radius R,
about the local mean 4. Take Mean of all points in
window, this becomes new local mean
![Page 10: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/10.jpg)
Algorithm 3
Given a set of points in a Cartesian plane 1. Select a Starting Point 2. This point becomes a local
mean. 3. Cast a window with radius R,
about the local mean 4. Take Mean of all points in
window, this becomes new local mean
5. Now mean is shifted
![Page 11: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/11.jpg)
Algorithm 3
Given a set of points in a Cartesian plane 1. Select a Starting Point 2. This point becomes a local
mean. 3. Cast a window with radius R,
about the local mean 4. Take Mean of all points in
window, this becomes new local mean
5. Now mean is shifted 6. Recast Window and Repeat
Process
![Page 12: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/12.jpg)
Algorithm 3
Given a set of points in a Cartesian plane 1. Select a Starting Point 2. This point becomes a local mean. 3. Cast a window with radius R,
about the local mean 4. Take Mean of all points in
window, this becomes new local mean
5. Now mean is shifted 6. Recast Window and Repeat
Process. 7. Eventually a Convergence is
reached and
![Page 13: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/13.jpg)
Formula
• Mean Shift – Is an algorithm for finding the local mode, or modes in a sample population. It is also known as a gradient finding algorithm when used with a Gaussian Kernel
Mean-Shift Formula Kernel: Gaussian
Weighted Mean
![Page 14: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/14.jpg)
Algorithm
Given a set of points in a Cartesian plane 1. Select a Starting Point 2. This point becomes a local mean. 3. Cast a window with radius R,
about the local mean 4. Take Mean of all points in
window, this becomes new local mean
5. Now mean is shifted 6. Recast Window and Repeat
Process. 7. Eventually a Convergence is
reached and
![Page 15: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/15.jpg)
Clustering Results
![Page 16: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/16.jpg)
Part II Variance Estimation
![Page 17: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/17.jpg)
Variance, Normal Distribution
*[1] All data above from wikipedia: http://en.wikipedia.org/wiki/Normal_distribution
[1]
![Page 18: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/18.jpg)
Mean Shift Assumes Equal Size Distributions
• What Are the true sizes of these clusters
• We must not assume all our clusters have the same distribution size in terms of variance.
![Page 19: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/19.jpg)
Variance Estimation and Displacement Vector
• The Diamonds Represent the displacement at iterations in the clustering
• The “Trail” is representative of the gradient finding process.
• The displacement trail can be used to determine the variance of an underlying distribution.
![Page 20: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/20.jpg)
Conceptual Idea
• Displacement Analysis:
– Is really a change to the process of clustering
– Normally we have a fixed window size.
![Page 21: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/21.jpg)
Conceptual Idea
• Displacement Analysis:
– The difference is when convergence is reached
– We increase the size of the window incrementally
– Re-Compute local mean
– Let converge
![Page 22: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/22.jpg)
Conceptual Idea
• Displacement Analysis:
– Keep repeating until displacement vector experiences a slight divergence & re-convergence
– This happens as window ingests statistical outliers to the distribution
![Page 23: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/23.jpg)
Two Normal Distributions
Distribution 1: Center 0,0 Variance 1
Distribution 2 Center 3,2
Variance 2.2
![Page 24: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/24.jpg)
Displacement Analysis
1. Plateau Corresponds to peak, indicates close variance approximation
2. Once bottom graph converges, top graph diverges
![Page 25: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/25.jpg)
Displacement Analysis
1. Window Size does not directly equal variance, must account for this.
2. This algorithm window, is tuned for variance in weight function.
3. This is not fool proof
![Page 26: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/26.jpg)
The End
![Page 27: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/27.jpg)
({[Back Up]})
![Page 28: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/28.jpg)
Displacement Analysis
During Normal Clustering Operation
When convergence is reached, we Dilate
window
![Page 29: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/29.jpg)
Back - Up
![Page 30: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/30.jpg)
Displacement Analysis
The window size reflects the variance size, and can be seen
in this plateau
![Page 31: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/31.jpg)
Displacement Analysis
Once the window reaches the true
distribution size the displacement
“peaks” and then “Drops”
![Page 32: Gaussian Shift (Mean Shift) Clustering and Variance ...web.williams.edu/Mathematics/sjmiller/public_html... · –Clustering •Technique & Theory •Part 2 –Variance Estimation](https://reader033.fdocuments.us/reader033/viewer/2022050216/5f622969f9a2bb09f912ccfb/html5/thumbnails/32.jpg)
Method Two – Group Cluster Centers