Image Modeling & Segmentation Aly Farag and Asem Ali Lecture #3.
-
Upload
kali-ashwell -
Category
Documents
-
view
213 -
download
0
Transcript of Image Modeling & Segmentation Aly Farag and Asem Ali Lecture #3.
Image Modeling &
SegmentationAly Farag and Asem Ali
Lecture #3
2
Parametric methods
These methods are useful when the underlying distribution is known in advance or is simple enough to be modeled by a simple distribution function or a mixture of such functions
The parametric model is very compact (low memory and CPU usage) where only few parameters need to fit.
The model’s parameters are estimated using these methods such as maximum likelihood estimation, Bayesian estimation and expectation maximization.
o A location parameter simply shifts the graph left or right on the horizontal
axis. o A scale parameter (>1) stretches or (<1 ) compress the pdf,
n
ii
i
n
i
nMLE
p
p
p
1
1
21
)|(logmaxarg
)|(logmaxarg
)|,,,(logmaxarg
x
x
xxx
3
Parametric methods (1- Maximum Likelihood Estimator: MLE)
Suppose that n samples x1, x2, … , xn are drawn independently and identically distributed (i.i.d.) ~ distribution φ(θ) with vector of parameters θ=(θ1,…. , θr)
Know: data samples, the distribution type Unknown : θ ?????
MLE Method estimates θ by maximizing the log likelihood of the data
By i.i.d.
monotonicity of log
To show the dependence of p on ɵ explicitly.
n
iiMLE
n
iiMLE
n
i
in
i
iT
nn
nLLL
1
22
1
14
2
21
2
)(11
0]2
)(
2,[],[
xx
xx
4
Parametric methods (1- Maximum Likelihood Estimator: MLE)Let
Then calculate
Find θ by letting
n
iipL
1
)|(log x
T
r
LLLL],,,[
21
0L
Example:
Suppose that n samples x1, x2, … , xn are drawn independently and identically distributed (i.i.d.)~ 1D N(μ,σ) find the MLE of μ and σ
n
i
in
i
xn
ii nepL
i
12
2
1
2
)(
1 2
)(
2
1log
2
1log)|(log)(
2
2
x
x
Matlab Demo
In some cases we can find a closed form for θ
Coin Example ………………..
i
n
ii
n
iiMLE nE
nE
nnEE xxx
111][
11
5
An estimator of a parameter is unbiased if the expected value of the estimate is the same as the true value of the parameters.
Parametric methods (1- Maximum Likelihood Estimator: MLE)
Example:
An estimator of a parameter is biased if the expected value of the estimate is different from the true value of the parameters.
Example:
2
1
2
12
1
2
1
2
11
22
11
............................................
))(12
(1
)1
(1
)(1
][
n
prooftheComplete
nnnE
nnE
nEE
n
i
n
jj
n
ijii
n
i
n
jji
n
iMLEiMLE
xxxx
xxx
n
iMLEiunbiased n 1
22 )(1
1 x doesn’t make much difference once n --> large
K
kkk xxp
1
;)(
6
What if there are distinct subpopulations in observed data?
Parametric methods
Example Pearson in 1894, tried to model the distribution of the ratio
between measurements of forehead and body length on crabs.
He used a two-component mixture.
It was hypothesized that the two-component structure was related to the possibility of this particular population of crabs evolving into two new subspecies
Mixture Model
The underlying density is assumed to have the form
The weights, Constrained
K
kkk
1
1,0
Components of the mixture are densities and are
parameterized by k
What is the difference between Mixture Model and the kernel-based estimator?
2
2
1
1 1,12
2
1,11
1
11 n
Ci
i
n
Ci
i
ii
nnxx
7
Parametric methods
Example Given that {xi, Ci1, Ci2} n samples (complete data) drawn i.i.d. two
normal distributions
xi observed value of ith instance
Ci1 and Ci2 indicate which of two normal distributions was used
to generate xi
Cij=1 if Cij was used to generate xi, 0 otherwise
MLE
2
2
1
1 1,1
22
2
22
1,1
21
1
21 )(
1)(
1 n
Ci
i
n
Ci
i
ii
nn xx
How can we estimate the parameters given incomplete data (don’t know Ci1 and Ci2 )?
8
Parametric methods (2- Expectation Maximization: EM)
The EM algorithm is a general method of finding the maximum-likelihood estimate of the parameters of an underlying distribution from a given data set when the data is incomplete or has missing values.
EM Algorithm:
Given initial parameters Θ0
Repeatedly
o Re-estimating expected values of hidden binary variables Cij
o Then recalculate the MLE of Θ using these expected values for the hidden variables Note:
EM unsupervised method, but MLE supervised
To use EM you must to know:
Number of classes K,
Parametric form of the distribution.
9
Illustrative example
complete incomplete
10
Illustrative example
)|,( cxp
11
Parametric methods (2- Expectation Maximization: EM)
Assume that a joint density function for complete data set
The EM algorithm first finds the expected value of the complete-data log-likelihood with respect to the unknown data C given the observed data x and the current parameter estimates Θi-1.
11 ,|)|,(log)|( ii xcxpEQ
The current parameters estimates that we used to evaluate the expectation
The new parameters that we optimize to maximize Q
The evaluation of this expectation is called the E-step of the algorithm
The second step (the M-step) of the EM algorithm is to maximize the expectation we computed in the first step.
)|(maxarg 1
ii Q
These two steps are repeated as necessary. Each iteration is guaranteed to increase the log likelihood and the algorithm is guaranteed to converge to a local maximum of the likelihood function.
K
kkk xxp
1
;)(
12
Parametric methods (2- Expectation Maximization: EM)
The mixture-density parameter estimation problem
],,,,,[ 11 KK
dcxcpcxpxcxpEQCc
iii
),|()|,(log,|)|,(log)|( 111
),|( 1
1
i
jj
n
jxcp
K
k
ikj
i
icj
i
ijj
xp
xpxcp
k
jjc
1
1
)|(
)|(),|(
Using Bayes’s rule, we can compute:
),|())|(log()|( 1
1 1
1
ij
K
k
n
jkjk
i xkpxpQ
Then
E-step
Grades Example ………………..
,
13
Parametric methods (2- Expectation Maximization: EM)
For some distributions, it is possible to get an analytical expressions for For example, if we assume d-dimensional Gaussian component distributions
n
j
ikj
ikj
n
jj
i
xkp
xkpx
k
1
1
1
1
),|(
),|(
n
j
ikj
n
j
Tij
ij
ikj
i
xkp
xxxkpkk
k
1
1
1
1
),|(
))()(,|(
n
j
ikj
i xkpnk
1
1),|(1
M-step
K
k
ikj
i
icj
i
ijj
xp
xpxcp
k
jjc
1
1
)|(
)|(),|(
E-step
0 50 100 150 200 2500
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
0.02
14
Parametric methods (2- Expectation Maximization: EM)
Example:
MATLAB demo