Point EstimationNotes of STAT 6205 by Dr. Fan

Overview• Section 6.1

• Point estimation

• Maximum likelihood estimation

• Methods of moments

• Sufficient statisticso Definition

o Exponential family

o Mean square error (how to choose an estimator)

6205-Ch6 2

Big Picture• Goal: To study the unknown distribution of a

population

• Method: Get a representative/random sample and use the information obtained in the sample to make statistical inference on the unknown features of the distribution

• Statistical Inference has two parts: o Estimation (of parameters)

o Hypothesis testing

• Estimation:o Point estimation: use a single value to estimate a parameter

o Interval estimation: find an interval covering the unknown parameter

6205-Ch6 3

Point Estimator• Biased/unbiased: an estimator is called unbiased if

its mean is equal to the parameter of estimate; otherwise, it is biased

• Example: X_bar is unbiased for estimating mu

6205-Ch6 4

Maximum Likelihood

Estimation• Given a random sample X1, X2, …, Xn from a

distribution f(x; θ) where θ is a (unknown) value in the parameter space Ω.

• Likelihood function vs. joint pdf

• Maximum Likelihood Estimator (m.l.e.) of θ, denoted as is the value θ which maximizes the likelihood function, given the sample X1, X2, …, Xn.

∏=

=n

iixfxL

1

);();( θθ

θ

6205-Ch6 5

Examples/Exercises• Problem 1: To estimate p, the true probability of heads

up for a given coin.

• Problem 2: Let X1, X2, …, Xn be a random sample from a Exp(mu) distribution. Find the m.l.e. of mu.

• Problem 3: Let X1, X2, …, Xn be a random sample from a Weibull(a=3,b) distribution. Find the m.l.e. of b.

• Problem 4: Let X1, X2, …, Xn be a random sample from a N(µ,σ^2) distribution. Find the m.l.e. of µ and σ.

• Problem 5: Let X1, X2, …, Xn be a random sample from a Weibull(a,b) distribution. Find the m.l.e. of a and b.

6205-Ch6 6

Method of Moments• Idea: Set population moments = sample moments

and solve for parameters

• Formula: When the parameter θ is r-dimensional, solve the following equations for θ:

6205-Ch6 7

∑=

==n

i

ki

k ,...r,knXXE1

21for /)(

Examples/ExercisesGiven a random sample from a population

• Problem 1: Find the m.m.e. of µ for a Exp(µ) population.

• Exercise 1: Find the m.m.e. of µ and σ for a N(µ,σ^2) population.

6205-Ch6 8

Sufficient Statistics• Idea: The “sufficient” statistic contains all

information about the unknown parameter; no other statistic can provide additional information as to the unknown parameter.

• If for any event A, P[A|Y=y] does not depend on the unknown parameter, then the statistic Y is called “sufficient” (for the unknown parameter).

• Any one-to-one mapping of a sufficient statistic Y is also sufficient.

• Sufficient statistics do not need to be estimators of the parameter.

6205-Ch6 9

Sufficient Statistics

6205-Ch6 10

Examples/ExercisesLet X1, X2, …, Xn be a random sample from f(x)

Problem: Let f be Poisson(a). Prove that

1. X-bar is sufficient for the parameter a

2. The m.l.e. of a is a function of the sufficient statistic

Exercise: Let f be Bin(n, p). Prove that X-bar is sufficient for p (n is known). Hint: find a sufficient statistic Y for p and then show that X-bar is a function of Y

6205-Ch6 11

Exponential Family

6205-Ch6 12

Examples/Exercises

Example 1: Find a sufficient statistic for p for Bin(n, p)

Example 2: Find a sufficient statistic for a for Poisson(a)

Exercise: Find a sufficient statistic for µ for Exp(µ)

6205-Ch6 13

Joint Sufficient Statistics

Example: Prove that X-bar and S^2 are joint sufficient statistics for µ and σ of N(µ, σ^2)

6205-Ch6 14

Application of Sufficience

6205-Ch6 15

ExampleConsider a Weibull distribution with parameter(a=2, b)

1) Find a sufficient statistic for b

2) Find an unbiased estimator which is a function of the sufficient statistic found in 1)

6205-Ch6 16

Good Estimator?• Criterion: mean square error

6205-Ch6 17

Example• Which of the following two estimator of variance is

better?

6205-Ch6 18