What is it? When would you use it? Why does it work? How do you implement it?
description
Transcript of What is it? When would you use it? Why does it work? How do you implement it?
What is it?
When would you use it?
Why does it work?
How do you implement it?
Where does it stand in relation to other methods?
EM algorithm reading group
Introduction & Motivation
Theory
Practical
Comparison with other methods
Expectation Maximization (EM)
• Iterative method for parameter estimation where you have missing data
• Has two steps: Expectation (E) and Maximization (M)
• Applicable to a wide range of problems
• Old idea (late 50’s) but formalized by Dempster, Laird and Rubin in 1977
• Subject of much investigation. See McLachlan & Krishnan book 1997.
Applications of EM (1)
• Fitting mixture models
Applications of EM (2)
• Probabilistic Latent Semantic Analysis (pLSA)– Technique from text community
P(w,d) P(w|z) P(z|d)
Z
WD
Z
D
W
Applications of EM (3)
• Learning parts and structure models
Applications of EM (4)
• Automatic segmentation of layers in video
http://www.psi.toronto.edu/images/figures/cutouts_vid.gif
Motivating example
0 1-2 -1-4 -3 4 52 3
Data:
Model:
Parameters:
OBJECTIVE: Fit mixture of Gaussian model with C=2 components
keep fixed
i.e. only estimate x
P(x|)
where
Likelihood functionLikelihood is a function of parameters, Probability is a function of r.v. x DIFFERENT TO LAST PLOT
Probabilistic modelImagine model generating data
Need to introduce label, z, for each data point
Label is called a latent variablealso called hidden, unobserved, missing
c
0 1-2 -1-4 -3 4 52 3
Simplifies the problem:if we knew the labels, we can decouple the components as estimate
parameters separately for each one
Intuition of EME-step: Compute a distribution on the labels of the points, using current parameters
M-step: Update parameters using current guess of label distribution.
E
E
M
M
E
Theory
Complete log-likelihood (CLL)
Log-likelihood [Incomplete log-likelihood (ILL)]
Expected complete log-likelihood (ECLL)
Some definitionsObserved data
Latent variables
Iteration index
Continuous I.I.D
Discrete 1 ... C
Use Jensen’sinequality
Lower bound on log-likelihood
AUXILIARY FUNCTION
Jensen’s InequalityJensen’s inequality:
For a real continuous concave function and
Equality holds when all x are the same
1. Definition of concavity. Consider where
then
2. By induction:
for
Recall key result : Auxiliary function is LOWER BOUND on likelihood
EM is alternating ascent
Alternately improve q then :
Is guaranteed to improve likelihood itself….
E-step: Choosing the optimal q(z|x,)
Turns out that q(z|x,) = p(z|x,t) is the best.
E-step: What do we actually compute?
nComponents x nPoints matrix (columns sum to 1):
Component 1
Component 2
Poi
nt 1
Poi
nt 2
Poi
nt 6
Responsibility of component for point :
M-Step
Entropy termECLL
Auxiliary function separates into ECLL and entropy term:
M-Step
From previous slide:
Recall definition of ECLL:
From E-step
Let’s see what happens for
Practical
InitializationMean of data + random offsetK-Means
TerminationMax # iterationslog-likelihood changeparameter change
ConvergenceLocal maximaAnnealed methods (DAEM)Birth/death process (SMEM)
Numerical issuesInject noise in covariance matrix to prevent blowupSingle point gives infinite likelihood
Number of componentsOpen problemMinimum description lengthBayesian approach
Practical issues
Local minima
Robustness of EM
What EM won’t do
Pick structure of model# componentsgraph structure
Find global maximum
Always have nice closed-form updatesoptimize within E/M step
Avoid computational problemssampling methods for computing expectations
Comparison with other methods
Why not use standard optimization methods?
• No step size
• Works directly in parameter space model, thus parameter constraints are obeyed
• Fits naturally into graphically model frame work
• Supposedly faster
In favour of EM:
Gradient
Newton
EM
Gradient
Newton
EM
Acknowledgements
Shameless stealing of figures and equations and explanations from:
Frank DellaertMichael JordanYair Weiss