Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP...
Transcript of Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP...
![Page 1: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/1.jpg)
Machine Learning:
Unsupervised Learning
Konstantin Tretyakov http://kt.era.ee
AACIMP 2012
August 2012
![Page 2: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/2.jpg)
So far…
AACIMP Summer School.
August, 2012
![Page 3: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/3.jpg)
So far…
AACIMP Summer School.
August, 2012
Optimization
Probability theory
Reasoning by analogy
![Page 4: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/4.jpg)
So far…
AACIMP Summer School.
August, 2012
Optimization Reasoning by analogy
Fermat’s theorem
Gradient methods
Batch & On-line
Probability theory
MLE,
MAP,
Bayesian Estimation
Risk Optimization
k-NN,
Kernel methods
![Page 5: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/5.jpg)
So far…
AACIMP Summer School.
August, 2012
Optimization Reasoning by analogy
Fermat’s theorem
Gradient methods
Batch & On-line
Probability theory
MLE,
MAP,
Bayesian Estimation
Risk Optimization
k-NN,
Kernel methods k-NN, Decision tree learning
Linear regression (OLS, RR),
Linear classification (SVM, Perceptron),
Bayes & Naïve Bayes
Kernel-<xxx>
![Page 6: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/6.jpg)
Today
AACIMP Summer School.
August, 2012
Optimization Reasoning by analogy
Fermat’s theorem
Gradient methods
Batch & On-line
Probability theory
MLE,
MAP,
Bayesian Estimation
Risk Optimization
k-NN,
Kernel methods k-NN, Decision tree learning
Linear regression (OLS, RR),
Linear classification (SVM, Perceptron),
Bayes & Naïve Bayes
Kernel-<xxx>
Unsupervised
Learning
![Page 7: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/7.jpg)
AACIMP Summer School.
August, 2012
![Page 8: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/8.jpg)
AACIMP Summer School.
August, 2012
![Page 9: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/9.jpg)
AACIMP Summer School.
August, 2012
![Page 10: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/10.jpg)
AACIMP Summer School.
August, 2012
![Page 11: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/11.jpg)
AACIMP Summer School.
August, 2012
![Page 12: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/12.jpg)
Data Mining
AACIMP Summer School.
August, 2012
Raw data
Pattern
Deviations
![Page 13: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/13.jpg)
Today
AACIMP Summer School.
August, 2012
Clustering Decomposition
![Page 14: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/14.jpg)
Quiz
Why would one need clustering?
AACIMP Summer School.
August, 2012
![Page 15: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/15.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
![Page 16: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/16.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
![Page 17: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/17.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
![Page 18: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/18.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
![Page 19: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/19.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
Complete linkage
![Page 20: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/20.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
Single linkage
![Page 21: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/21.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
Average linkage
![Page 22: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/22.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
Ward linkage
𝜎2
![Page 23: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/23.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
![Page 24: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/24.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
![Page 25: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/25.jpg)
Hierarchical clustering
AACIMP Summer School.
August, 2012
![Page 26: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/26.jpg)
Partitional vs Hierarchical
AACIMP Summer School.
August, 2012 Slide © M.Kull, http://courses.cs.ut.ee/2012/ml/
![Page 27: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/27.jpg)
K-means
AACIMP Summer School.
August, 2012
![Page 28: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/28.jpg)
K-means
AACIMP Summer School.
August, 2012
![Page 29: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/29.jpg)
K-means
AACIMP Summer School.
August, 2012
![Page 30: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/30.jpg)
K-means
AACIMP Summer School.
August, 2012
![Page 31: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/31.jpg)
K-means
AACIMP Summer School.
August, 2012
![Page 32: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/32.jpg)
K-means
AACIMP Summer School.
August, 2012
![Page 33: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/33.jpg)
K-means
AACIMP Summer School.
August, 2012
argmin𝒄𝟏,…,𝒄𝑲 𝒙𝒊 − 𝒄𝐜𝐥𝐨𝐬𝐞𝐬𝐭_𝐭𝐨 (𝒊)𝟐
𝒊
![Page 34: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/34.jpg)
K-means
AACIMP Summer School.
August, 2012
argmin𝒄𝟏,…,𝒄𝑲 𝒙𝒊 − 𝒄𝐜𝐥𝐨𝐬𝐞𝐬𝐭_𝐭𝐨(𝒊)𝟐
𝒊
• Need to find cluster centers 𝒄𝒌. 𝒄𝟏 = ? , 𝒄𝟐 = ? , … , 𝒄𝑲 =?
![Page 35: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/35.jpg)
K-means
AACIMP Summer School.
August, 2012
argmin𝒄𝟏,…,𝒄𝑲 𝒙𝒊 − 𝒄𝐜𝐥𝐨𝐬𝐞𝐬𝐭_𝐭𝐨(𝒊)𝟐
𝒊
• Need to find cluster centers 𝒄𝒌. 𝒄𝟏 = ? , 𝒄𝟐 = ? , … , 𝒄𝑲 =?
• Introduce latent variables (one for each 𝒙𝒊) 𝒂𝒊 = 𝐜𝐥𝐨𝐬𝐞𝐬𝐭_𝐜𝐥𝐮𝐬𝐭𝐞𝐫_𝐜𝐞𝐧𝐭𝐞𝐫(𝒊) 𝒂𝟏 =? , 𝒂𝟐 =? , 𝒂𝟑 =? ,… , 𝒂𝒏 =?
![Page 36: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/36.jpg)
K-means
AACIMP Summer School.
August, 2012
argmin𝒄𝟏,…,𝒄𝑲 𝒙𝒊 − 𝒄𝐜𝐥𝐨𝐬𝐞𝐬𝐭_𝐭𝐨(𝒊)𝟐
𝒊
• For fixed 𝒄𝒌 we can find optimal 𝒂𝒊
• For fixed 𝒂𝒊 we can find optimal 𝒄𝒌.
• Iterate to convergence.
![Page 37: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/37.jpg)
Fuzzy vs Hard
AACIMP Summer School.
August, 2012 Slide © M.Kull, http://courses.cs.ut.ee/2012/ml/
![Page 38: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/38.jpg)
Gaussian Mixture Modeling
AACIMP Summer School.
August, 2012
𝑿 ∼ [𝑁 𝝁1, 𝜎12 or 𝑁 𝝁2, 𝜎2
2 ]
Given 𝑿, estimate 𝝁𝒊, 𝝈𝒊𝟐
![Page 39: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/39.jpg)
Gaussian Mixture Modeling
AACIMP Summer School.
August, 2012
𝑿 ∼ [𝑁 𝝁1, 𝜎12 or 𝑁 𝝁2, 𝜎2
2 ]
Given 𝑿, estimate 𝝁𝒊, 𝝈𝒊𝟐
MLE
![Page 40: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/40.jpg)
Gaussian Mixture Modeling
AACIMP Summer School.
August, 2012
𝑿 ∼ [𝑁 𝝁1, 𝜎12 or 𝑁 𝝁2, 𝜎2
2 ]
Given 𝑿, estimate 𝝁𝒊, 𝝈𝒊𝟐
MLE
Expectation-Maximization (EM)
![Page 41: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/41.jpg)
SKLearn’s Clustering
from sklearn.cluster
import
Ward,
KMeans,
DBScan,
MeanShift,
SpectralClustering,
AffinityPropagation
AACIMP Summer School.
August, 2012
![Page 42: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/42.jpg)
SKLearn’s Clustering
from sklearn.cluster
import
Ward,
KMeans,
DBScan,
MeanShift,
SpectralClustering,
AffinityPropagation
AACIMP Summer School.
August, 2012
Use feature vectors
Use distance matrix
![Page 43: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/43.jpg)
Quiz
Fuzzy clustering means that _______
K-means finds a set of cluster centers, which
have the smallest ______________
K-means can get stuck in a local minimum
(Y/N)?
AACIMP Summer School.
August, 2012
![Page 44: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/44.jpg)
Today
AACIMP Summer School.
August, 2012
Clustering Decomposition
![Page 45: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/45.jpg)
Canonical basis
𝑥1𝑥2= 𝛼10+ 𝛽01
AACIMP Summer School.
August, 2012
![Page 46: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/46.jpg)
Alternative basis
𝑥1𝑥2= 𝛼
0.9−0.1
+ 𝛽0.10.9
AACIMP Summer School.
August, 2012
![Page 47: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/47.jpg)
Alternative basis
𝑥1𝑥2= 𝛼
0.4−0.6
+ 𝛽0.60.4
AACIMP Summer School.
August, 2012
![Page 48: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/48.jpg)
Linear Decomposition
𝑥1𝑥2𝑥3𝑥4…𝑥100000
= 𝛼1
1000⋮0
+ 𝛼2
0100⋮0
+⋯+ 𝛼𝑚
0000⋮1
AACIMP Summer School.
August, 2012
![Page 49: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/49.jpg)
Linear Decomposition
𝑥1𝑥2𝑥3𝑥4…𝑥100000
= 𝛼1
1000⋮0
+ 𝛼2
0100⋮0
+⋯+ 𝛼𝑚
0000⋮1
AACIMP Summer School.
August, 2012
![Page 50: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/50.jpg)
Linear Decomposition
𝑥1𝑥2𝑥3𝑥4…𝑥100000
= 𝛼1
0.00.10.10.2⋮0.0
+ 𝛼2
0.30.20.20.1⋮0.3
+ 𝛼𝑚
0.10.10.10.0⋮0.0
AACIMP Summer School.
August, 2012
![Page 51: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/51.jpg)
Linear Decomposition
𝑥1𝑥2𝑥3𝑥4…𝑥100000
= 𝛼1
0.00.10.10.2⋮0.0
+ 𝛼2
0.30.20.20.1⋮0.3
+ 𝛼𝑚
0.10.10.10.0⋮0.0
AACIMP Summer School.
August, 2012
![Page 52: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/52.jpg)
Linear Decomposition
𝑥1𝑥2𝑥3𝑥4…𝑥100000
= 𝛼1
1000⋮0
+ 𝛼2
0100⋮0
+⋯+ 𝛼𝑚
0000⋮1
AACIMP Summer School.
August, 2012
![Page 53: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/53.jpg)
Linear Decomposition
𝑥1𝑥2𝑥3𝑥4…𝑥100000
= 𝛼1
0.00.10.10.2⋮0.0
+ 𝛼2
0.30.20.20.1⋮0.3
+ 𝛼𝑚
0.10.10.10.0⋮0.0
AACIMP Summer School.
August, 2012
![Page 54: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/54.jpg)
Linear Decomposition
𝑥1𝑥2𝑥3𝑥4…𝑥100000
= 𝛼1
0.00.10.10.2⋮0.0
+ 𝛼2
0.30.20.20.1⋮0.3
+ 𝛼𝑚
0.10.10.10.0⋮0.0
AACIMP Summer School.
August, 2012
![Page 55: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/55.jpg)
Linear Decomposition
𝒙 = 𝛼1𝒗𝟏 + 𝛼2𝒗𝟐 +⋯+ 𝛼𝑚𝒗𝒎
AACIMP Summer School.
August, 2012
![Page 56: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/56.jpg)
Linear Decomposition
𝒙 = 𝛼1𝒗𝟏 + 𝛼2𝒗𝟐 +⋯+ 𝛼𝑚𝒗𝒎
𝒙 =⋮ ⋮ … ⋮𝒗𝟏⋮𝒗𝟐⋮…𝒗𝒎⋮
𝛼1𝛼2⋮𝛼𝑚
AACIMP Summer School.
August, 2012
![Page 57: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/57.jpg)
Linear Decomposition
𝒙 = 𝛼1𝒗𝟏 + 𝛼2𝒗𝟐 +⋯+ 𝛼𝑚𝒗𝒎
𝒙 = 𝑽𝜶
AACIMP Summer School.
August, 2012
![Page 58: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/58.jpg)
Linear Decomposition
𝒙 = 𝛼1𝒗𝟏 + 𝛼2𝒗𝟐 +⋯+ 𝛼𝑚𝒗𝒎
𝒙 = 𝑽𝜶 𝜶 = ?
AACIMP Summer School.
August, 2012
![Page 59: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/59.jpg)
Linear Decomposition
𝒙 = 𝛼1𝒗𝟏 + 𝛼2𝒗𝟐 +⋯+ 𝛼𝑚𝒗𝒎
𝒙 = 𝑽𝜶 𝜶 = 𝑽+𝒙
AACIMP Summer School.
August, 2012
![Page 60: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/60.jpg)
How do we find a good basis?
AACIMP Summer School.
August, 2012
![Page 61: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/61.jpg)
Linear projection
AACIMP Summer School.
August, 2012
![Page 62: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/62.jpg)
Linear projection
AACIMP Summer School.
August, 2012
![Page 63: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/63.jpg)
Linear projection
AACIMP Summer School.
August, 2012
![Page 64: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/64.jpg)
Linear projection
AACIMP Summer School.
August, 2012
![Page 65: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/65.jpg)
Idea: Maximize projection variance
For a point 𝒙𝑖 and a unit basis vector 𝒗 the
length of projection of 𝒙𝑖 onto 𝒗 is given by
𝑝 = 𝒗, 𝒙𝑖 = 𝒗𝑇𝒙𝑖
AACIMP Summer School.
August, 2012
𝒗
𝒙
𝑝
![Page 66: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/66.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
AACIMP Summer School.
August, 2012
![Page 67: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/67.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
𝜎𝒗2 =1
𝑛 𝑝𝑖 − 𝑝
2
𝑖
AACIMP Summer School.
August, 2012
![Page 68: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/68.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
𝜎𝒗2 =1
𝑛 𝑝𝑖 − 𝑝
2
𝑖
𝒗 = argmax𝒗 𝜎𝒗2
AACIMP Summer School.
August, 2012
![Page 69: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/69.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
𝜎𝒗2 =1
𝑛 𝑝𝑖 − 𝑝
2
𝑖
AACIMP Summer School.
August, 2012
Pre-center data, so that 𝑝 = 𝒗𝑇𝒙 = 0
![Page 70: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/70.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
𝜎𝒗2 =1
𝑛 𝑝𝑖
2
𝑖
AACIMP Summer School.
August, 2012
Pre-center data, so that 𝑝 = 𝒗𝑇𝒙 = 0
![Page 71: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/71.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
𝜎𝒗2 =1
𝑛 𝑝𝑖
2
𝑖
=1
𝑛𝒑 2
AACIMP Summer School.
August, 2012
![Page 72: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/72.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
𝜎𝒗2 =1
𝑛 𝑝𝑖
2
𝑖
=1
𝑛𝒑 2
=1
𝑛𝑿𝒗 2
AACIMP Summer School.
August, 2012
![Page 73: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/73.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
𝜎𝒗2 = ⋯
=1
𝑛𝑿𝒗 2 =
1
𝑛𝑿𝒗 𝑇 𝑿𝒗
=1
𝑛𝒗𝑇𝑿𝑇𝑿𝒗 = 𝒗𝑇𝚺𝒗
AACIMP Summer School.
August, 2012
![Page 74: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/74.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
𝜎𝒗2 = 𝒗𝑇𝚺𝒗
AACIMP Summer School.
August, 2012
![Page 75: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/75.jpg)
Projection variance
𝑝𝑖 = 𝒗𝑇𝒙𝑖
𝜎𝒗2 = 𝒗𝑇𝚺𝒗
AACIMP Summer School.
August, 2012
Data covariance matrix
𝑿𝑻𝑿
![Page 76: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/76.jpg)
Objective function
argmax𝒗 𝒗𝑇𝚺𝒗
𝑠. 𝑡. 𝒗 2 = 1
AACIMP Summer School.
August, 2012
![Page 77: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/77.jpg)
Optimization
argmax𝒗 𝒗𝑇𝚺𝒗
𝑠. 𝑡. 𝒗 2 = 1
𝚺𝒗 = 𝜆𝒗
AACIMP Summer School.
August, 2012
Method of Lagrange multipliers…
![Page 78: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/78.jpg)
Optimization
argmax𝒗 𝒗𝑇𝚺𝒗
𝑠. 𝑡. 𝒗 2 = 1
𝚺𝒗 = 𝜆𝒗
AACIMP Summer School.
August, 2012
Method of Lagrange multipliers…
Eigenvector of 𝚺 Eigenvalue
![Page 79: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/79.jpg)
Example
Xc = X – mean(X, axis=0)
Sigma = Xc.T * Xc / n
(= cov(Xc, rowvar=0))
lambdas, vs = eigh(Sigma)
AACIMP Summer School.
August, 2012
![Page 80: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/80.jpg)
Example
AACIMP Summer School.
August, 2012
𝜆1 = 0.8
𝜆2 = 28.9
![Page 81: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/81.jpg)
Example
AACIMP Summer School.
August, 2012
𝜆1 = 0.8
𝜆2 = 28.9
𝜎𝑖2 = 𝒗𝑖
𝑇𝚺𝒗𝑖 = 𝒗𝑖𝑇𝜆𝑖𝒗𝑖 = 𝜆𝑖 𝒗𝑖
2 = 𝜆𝑖
![Page 82: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/82.jpg)
Example
AACIMP Summer School.
August, 2012
𝜆2 = 28.9
![Page 83: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/83.jpg)
Example
AACIMP Summer School.
August, 2012
𝜆2 = 28.9
![Page 84: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/84.jpg)
Example
AACIMP Summer School.
August, 2012
𝜆1 = 0.8
𝜆2 = 28.9
![Page 85: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/85.jpg)
Principal Components Analysis
AACIMP Summer School.
August, 2012
Principal components are the
eigenvectors of the covariance matrix.
𝑽, 𝝀 = 𝐞𝐢𝐠(𝚺)
![Page 86: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/86.jpg)
Principal Components Analysis
AACIMP Summer School.
August, 2012
Principal components are the
eigenvectors of the covariance matrix.
𝑽, 𝝀 = 𝐞𝐢𝐠(𝚺)
For each PC, the corresponding eigenvalue
𝜆𝑖 shows the amount of variance
explained by the component.
![Page 87: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/87.jpg)
Principal Components Analysis
AACIMP Summer School.
August, 2012
Eigenvalue spectrum of 𝚺
![Page 88: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/88.jpg)
Principal Components Analysis
AACIMP Summer School.
August, 2012
Data projection onto PC 𝑖: 𝒑 = 𝑿𝒗𝑖
Data reconstruction from PC coordinates:
𝑿proj𝑽∗𝑇 = 𝑿
Data projection onto multiple PCs:
𝑿proj = 𝑿𝑽∗
![Page 89: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/89.jpg)
SKLearn’s PCA
from sklearn.decomposition
import PCA
model = PCA(n_components=2)
model.fit(X)
X_t = model.transform(X)
model.components_[1,:]
AACIMP Summer School.
August, 2012
![Page 90: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/90.jpg)
SKLearn’s PCA
from sklearn.decomposition
import
PCA,
SparsePCA,
ProbabilisticPCA,
KernelPCA,
FastICA,
NMF,
DictionaryLearning,
...
AACIMP Summer School.
August, 2012
![Page 91: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/91.jpg)
PCA: Geometric intuition
𝑿 ∼ 𝑁 0,1
AACIMP Summer School.
August, 2012
![Page 92: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/92.jpg)
PCA: Geometric intuition
𝑿 ∼ 𝑁 0,1
𝑿′ = 𝑿5 00 0.9
AACIMP Summer School.
August, 2012
![Page 93: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/93.jpg)
PCA: Geometric intuition
𝑿 ∼ 𝑁 0,1
𝑿′ = 𝑿5 00 0.9
𝑿′′
= 𝑿′cos 0.8 − sin 0.8sin 0.8 cos 0.8
AACIMP Summer School.
August, 2012
![Page 94: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/94.jpg)
PCA: Geometric intuition
𝑿 ∼ 𝑁 0,1
𝑿′′ = 𝑿 ⋅ 𝑫 ⋅ 𝑹
AACIMP Summer School.
August, 2012
![Page 95: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/95.jpg)
PCA: Geometric intuition
𝑿 ∼ 𝑁 0,1
𝑿′′ = 𝑿 ⋅ 𝑫 ⋅ 𝑹
𝑿′′ 𝑇 𝑿′′
= 𝑿𝑫𝑹 𝑇(𝑿𝑫𝑹)
AACIMP Summer School.
August, 2012
![Page 96: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/96.jpg)
PCA: Geometric intuition
𝑿 ∼ 𝑁 0,1
𝑿′′ = 𝑿 ⋅ 𝑫 ⋅ 𝑹
𝑿′′ 𝑇 𝑿′′
= 𝑿𝑫𝑹 𝑇(𝑿𝑫𝑹) = 𝑹𝑇𝑫𝑇𝑿𝑇𝑿𝑫𝑹
AACIMP Summer School.
August, 2012
![Page 97: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/97.jpg)
PCA: Geometric intuition
𝑿 ∼ 𝑁 0,1
𝑿′′ = 𝑿 ⋅ 𝑫 ⋅ 𝑹
𝑿′′ 𝑇 𝑿′′
= 𝑿𝑫𝑹 𝑇(𝑿𝑫𝑹) = 𝑹𝑇𝑫𝑇𝑿𝑇𝑿𝑫𝑹 = 𝑹𝑇𝑫2𝑹
AACIMP Summer School.
August, 2012
![Page 98: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/98.jpg)
PCA: Geometric intuition
𝑿 ∼ 𝑁 0,1
𝑿′′ = 𝑿 ⋅ 𝑫 ⋅ 𝑹
𝑿′′ 𝑇 𝑿′′
= 𝑿𝑫𝑹 𝑇(𝑿𝑫𝑹) = 𝑹𝑇𝑫𝑇𝑿𝑇𝑿𝑫𝑹 = 𝑹𝑇𝑫2𝑹
AACIMP Summer School.
August, 2012
𝚺 = 𝑽𝚲𝑽𝑇 (𝑽 = 𝑹𝑻, 𝚲 = 𝐃2)
![Page 99: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/99.jpg)
PCA: Geometric intuition
𝑿 ∼ 𝑁 0,1
𝑿′′ = 𝑿 ⋅ 𝑫 ⋅ 𝑹
𝑿′′ 𝑇 𝑿′′
= 𝑿𝑫𝑹 𝑇(𝑿𝑫𝑹) = 𝑹𝑇𝑫𝑇𝑿𝑇𝑿𝑫𝑹 = 𝑹𝑇𝑫2𝑹
AACIMP Summer School.
August, 2012
𝚺 = 𝑽𝚲𝑽𝑇 (𝑽 = 𝑹𝑻, 𝚲 = 𝐃2)
𝑿′′𝑹𝑇 = 𝑿𝑫
𝑿′′𝑽 = 𝑿𝐩𝐫𝐨𝐣
![Page 100: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/100.jpg)
AACIMP Summer School.
August, 2012
![Page 101: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/101.jpg)
AACIMP Summer School.
August, 2012
![Page 102: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/102.jpg)
Quiz
Principal components are ___________ of
the ___________ matrix.
Eigenvalue spectrum shows how much
___________ is explained by each ______.
If 𝚺 = 𝑽𝚲𝑽𝑇, then
𝑿𝐩𝐫𝐨𝐣 = _______
AACIMP Summer School.
August, 2012
![Page 103: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/103.jpg)
Conclusion
AACIMP Summer School.
August, 2012
![Page 104: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/104.jpg)
Conclusion
AACIMP Summer School.
August, 2012
Statistical Learning Theory,
PAC-theory
Semi-supervised learning,
Active learning,
Reinforcement learning,
Multi-instance learning,
Deep learning
Graphical models,
Neural networks,
Fuzzy sets,
Association rules
Computer vision,
Natural language processing,
Information Retrieval,
Music & Video processing,
Bioinformatics,
Physics,
Robotics,
Finance & Economics,
…
![Page 105: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/105.jpg)
Where to go next
Books: “The Elements of Statistical Learning” (Hastie & Tibshirani)
“Pattern Recognition and Machine Learning” (Bishop)
“Kernel Methods for Pattern Analysis” (Shawe-Taylor &
Cristianini)
On-line materials:
http://videolectures.net
+ Coursera, Udacity, edX
Tools:
Python, R, RapidMiner, Weka, Matlab, Mathematica, …
AACIMP Summer School.
August, 2012
![Page 106: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/106.jpg)
Where to go next
http://kt.era.ee/aacimp
AACIMP Summer School.
August, 2012
![Page 107: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/107.jpg)
AACIMP Summer School.
August, 2012
![Page 108: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/108.jpg)
AACIMP Summer School.
August, 2012
![Page 109: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/109.jpg)
AACIMP Summer School.
August, 2012
![Page 110: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/110.jpg)
AACIMP Summer School.
August, 2012
![Page 111: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/111.jpg)
AACIMP Summer School.
August, 2012
![Page 112: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/112.jpg)
AACIMP Summer School.
August, 2012
![Page 113: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/113.jpg)
AACIMP Summer School.
August, 2012
![Page 114: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/114.jpg)
Science
AACIMP Summer School.
August, 2012
Observation
Analysis
Generalization
![Page 115: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/115.jpg)
AACIMP Summer School.
August, 2012
![Page 116: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/116.jpg)
Why?
AACIMP Summer School.
August, 2012
![Page 117: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/117.jpg)
Why?
AACIMP Summer School.
August, 2012
![Page 118: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/118.jpg)
Contemporary Science
AACIMP Summer School.
August, 2012
Observation
Analysis
Generalization
![Page 119: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/119.jpg)
Contemporary Science
AACIMP Summer School.
August, 2012
Analysis
Generalization
CS & EE
![Page 120: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/120.jpg)
Contemporary Science
AACIMP Summer School.
August, 2012
![Page 121: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/121.jpg)
Contemporary Science
AACIMP Summer School.
August, 2012
![Page 122: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/122.jpg)
Contemporary Science
AACIMP Summer School.
August, 2012
![Page 123: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/123.jpg)
Contemporary Science
AACIMP Summer School.
August, 2012
Analysis
Generalization
CS & EE
![Page 124: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/124.jpg)
Contemporary Science
AACIMP Summer School.
August, 2012
Generalization
CS & EE
ML
![Page 125: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/125.jpg)
Contemporary Science
AACIMP Summer School.
August, 2012
CS & EE
ML
![Page 126: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/126.jpg)
AACIMP Summer School.
August, 2012
![Page 127: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/127.jpg)
AACIMP Summer School.
August, 2012
![Page 128: Machine Learning: Unsupervised Learningkt.era.ee/lectures/aacimp2012/slides5.pdf · Today AACIMP Summer School. August, 2012 Optimization Reasoning by analogy Fermat’s theorem Gradient](https://reader034.fdocuments.us/reader034/viewer/2022050315/5f773de0788eec235474e673/html5/thumbnails/128.jpg)
AACIMP Summer School.
August, 2012
Thank You!