Estimation of parameters

Maximum likelihood principle

What has happened was most likely

Maximize likelihood wrt parameters to obtain estimates

n

ii

n

xfL

xxxf

parameterssample

parameterseofthesamplLikelihood

1

21

)|()(

)|,,,(

]|Pr[

)(

Examples

Binomial distribution

kNkk pp

k

NpL

)1(

)1ln()(lnln pkNpkk

Nl

Observations: k successes in N Bernoulli trials

01

p

kN

p

k

dp

dl

N

kp ˆ

Poisson distribution

N

i i

kN

keL

i

1 )!(

Observations: k1, k2, …, kN

)...(ln 21 NkkkNl

Nkkk

Nd

dl

...21

N

kkk N

...ˆ 21

Normal distributionObservations: X1, X2, …,XN

N

i

iX

NNeL 1

2)(2

1

)2(

11

N

i

iXNl

1

2

2

1ln

N

i

iX

d

dl1

N

iiX

N 1

1̂

N

i

iXN

d

dl1

3

2

N

iiX

N 1

22 ˆ1

ˆ

Exponential distributionObservations: X1, X2, …,XN

N

iiXa

N eaL `1

N

iiXaaNl

`1ln

N

iiX

a

N

da

dl`1

N

iiX

Na

`1

ˆ

Moment estimators

),( axp

N

iiX

NaE

1

1)(

For our examples moment estimators = maximum likelihood

estimators

Are they always the same ?

No

Uniform distributionCauchy distribution

Unbiased and biased estimators

aaxp ˆ ),(

aaE )ˆ(

Variance of estimatorsminimum variance estimators