Week 7 Annotated

ACTL2002/ACTL5101 Probability and Statistics: Week 7

ACTL2002/ACTL5101 Probability and Statistics

c© Katja Ignatieva

School of Risk and Actuarial StudiesAustralian School of Business

University of New South Wales

[email protected]

Week 7Probability: Week 1 Week 2 Week 3 Week 4

Estimation: Week 5 Week 6 Review

Hypothesis testing: Week 8 Week 9

Linear regression: Week 10 Week 11 Week 12

Video lectures: Week 1 VL Week 2 VL Week 3 VL Week 4 VL Week 5 VL

mailto:[email protected]


2101/2155


Last six weeks

Introduction to probability;

Moments: (non)-central moments, mean, variance (standarddeviation), skewness & kurtosis;

Special univariate (parametric) distributions (discrete &continue);

Joint distributions;

Convergence; with applications LLN & CLT;

Estimators (MME, MLE, and Bayesian);

Evaluation of estimators;

Interval estimation.2102/2155


This & next weekHypothesis testing:

- Null hypothesis:Selection of the null hypothesis;

Simple v.s. composite.

- Statistical tests;

- Rejection region;Neyman-Pearson Lemma;

Uniformly most powerful;

Likelihood ratio test.

- Value of test;

- Accept the null or reject the null hypothesis.

Properties of hypothesis testing.- Type I and II errors and power;

- p-value.

Special tests and other testing methods.2103/2155


Introduction in Hypothesis testing

Introduction

Hypothesis testing

Introduction in Hypothesis testingIntroduction

Selection of null hypothesisThe first stepExample: testing proportions

Statistical testStatistical test

Rejection regionBest critical regionNeyman-Pearson LemmaNeyman-Pearson Lemma: examples and exercisesUniformly most powerful (UMP)Example: Uniformly most powerful (UMP)Generalized likelihood ratio testAsymptotic distribution for the GLR testExample: generalized likelihood ratio test

SummarySummary



Introduction

IntroductionFormal means of distinguishing between probabilitydistributions using sample data.

Examples:- Are the means of two normal distributions with the same

variance equal?

- Are the variances different?

- Is the parameter of the distribution equal to a constant?

Neyman-Pearson approach: null hypothesis H0; alternativehypothesis HA or H1.

We have different types of hypothesis:- Simple hypotheses: completely specify the probability

distribution.

- Composite hypotheses: probability distribution not completelyspecified.2104/2155



Introduction

Hypothesis testingIn hypothesis testing the choice is either reject or accept thehypothesis. Assume that X1,X2, . . . ,Xn are from a randomsample with density f (x ; θ).

A statistical hypothesis is defined to be a conjecture or anassertion about the distribution of a random variable(s). If thestatistical hypothesis completely specifies the distribution, it iscalled simple. Otherwise, it is called composite.

A statistical test of a statistical hypothesis, also called teststatistic, T is a rule or procedure for deciding whether toreject T .

The rejection region, or more commonly called the criticalregion, of a test T is the set of all (x1, x2, . . . , xn) for whichyou would reject the statistical hypothesis. This is usuallydenoted by C ?.2105/2155



Introduction

Hypothesis testing procedure

Testing procedure: When testing a hypothesis use the followingsteps:

i. Define a statistical hypothesis.Note that this includes a confidence level (α);

ii. Define the test statistic T (using past weeks knowledge);

iii. Determine the rejection region C ?;

iv. Calculate the value of the statistical test, given observed data(x1, . . . , xn);

v. Accept or reject H0.Note: we assume that H0 is true when testing! (see Type Iand Type II errors)

2106/2155


Selection of null hypothesis

The first step

Hypothesis testing





SummarySummary



The first step

Hypothesis testingNull Hypothesis H0: the hypothesis being tested.

Alternative Hypothesis Ha: the hypothesis accepted if H0 isrejected.

Results of Decision Making

HypothesisDecision True False

Reject H0 Type I Error=α No Error

Do not reject H0 No Error Type II Error=β

Type I, II errors: see properties of hypothesis testing (nextlecture).

We can only fix α, therefore the choice of the null isimportant.

2107/2155



The first step

Illustration Type I and Type II errors

Consider the following test:

H0 : no correlation between insurance claims Australia and US

versus

Ha : the correlation is positive

If you reject and in fact the there is positive correlation, thenno error would be committed.

On the other hand, if you reject but there is no correlation,then an error called a Type I error would have been committedand the cost might be that the insurer must hold more capital.

If you do not reject the null and later found there is positivecorrelation, this is called a Type II error and the cost would bepotential default due to too few reserves.

2108/2155



The first step

Selection of null hypothesisHow to select the Null hypothesis (step i.)?

- One way: select the simpler of the two hypotheses as the nullhypothesis;

- The economic method: The null hypothesis is the hypothesiswhere the consequences of incorrectly rejecting the nullhypothesis are largest;

- The statistical method: select the null as something that youwish to conclusively disprove.

The level of significance of the test can be determined from:

Pr (T ∈ C ? |H0 ) = α,

where T is a test statistic and C ? is the rejection region (seelater).

Confidence level α is the probability that you reject H0 giventhat H0 is true.

2109/2155



The first step

Simple and composite hypothesis

Recall from introduction, two kind of hypothesis:

If X ∼ f (x ; θ), a statistical hypothesis is a statement aboutthe distribution of X . If the hypothesis completely specifiesf (x ; θ), then it is referred to as a simple hypothesis otherwiseit is called composite.

2110/2155



The first step

Simple and composite hypothesisWe consider three cases:

- Simple null v.s. simple alternative;Example:H0 : θ = θ0 v.s. H1 : θ = θ1, for given constants θ0 and θ1.

- Simple null v.s. composite alternative;Example:H0 : θ = θ0 v.s. H1 : θ ∈ ω1, for given constant θ0 and set ω1.Where usually the alternative is of the form:

- H1 : θ 6= θ0;- H1 : θ < θ0;- H1 : θ > θ0.

- Composite null v.s. composite alternative;Example:H0 : θ ∈ ω0 v.s. H1 : θ ∈ ω1, for given sets ω0 and ω1.Where usually the null and alternative are of the form:

- H0 : θ ≤ θ0 vs H1 : θ > θ0;- H0 : θ ≥ θ0 vs H1 : θ < θ0.2111/2155



Example: testing proportions

Hypothesis testing





SummarySummary





An insurer investigates whether climate change has already an(positive or negative) effect on the probability that insuredlodge a claim.

Due to the Law of Large Numbers the insurer has a good(almost certain) estimate of the probability of lodging a claimbefore the climate change (say before T − 5). He knows thatthe probability was p0 = 0.1%.

The last five years the number of claims were 5 and thenumber of insurance contracts 2500.

Preform the test the insurer is going to make, with asignificance level of 5%.

2112/2155




Example: testing proportionsi. Consider the hypothesis test:

H0 : p = p0 v.s. Ha : p 6= p0,

with a random sample X1,X2, . . . ,Xn from a Bernoulli (p)distribution.

ii. We know that the best unbiased point estimator of p is:

p =1

n

n∑

k=1

Xk .

From central limit theorem (see week 6), we know that:

p − E [p]√Var (p)

=p − p√p (1− p)

n

,

has an approximate standard normal distribution.2113/2155




np0 np0 + z1−α/2

√np0(1 − p0)np0 − z1−α/2

√np0(1 − p0)

f(x|H0) →

When∑

Xi is in the blue shaded area ⇒ likely that p 6= p0 ⇒ reject H0.2114/2155




Under the null hypothesis H0 : p = p0, the observed teststatistic (know this from last week CI properties) is:

T =p − p0√p0 (1− p0)

n

,

where T is standard normal distributed.

iii. We then reject the null hypothesis if:

T < −z1−α/2 or T > z1−α/2

at a level of significance α. (More on this in the remainder ofthe lecture)

iv. We have p0 = 0.001, p = 0.002, n = 2500 thus T = 1.58193.

v. Do not reject H0. Question: Comment on the test.

Test uses: Bin(n, p)d→N(np, np(1− p)) which is good if

np →∞, but n · p0 = 2.5!2115/2155


Statistical test

Statistical test

Hypothesis testing





SummarySummary


Statistical test

Statistical test

Step ii. Define a statistical test.

For this we can use our previous obtained knowledge;

Recall from week 5: Central limit theorem.

Recall from week 5: Maximum likelihood estimator;

Recall from week 6: CI for Maximum likelihood estimates;

Recall from week 6: Interval estimation using confidenceintervals.

Assume null hypothesis is true, i.e., H0 defines the populationparameters, which are being tested.

2116/2155


Rejection region

Best critical region

Hypothesis testing





SummarySummary


Rejection region


The critical region for a test of hypothesis is the subset of thesample space that corresponds to rejecting the null hypothesis.

Example: Consider the following region to reject the nullhypothesis:

- Let X ∼ Bin(n, p), with n = 8 and unknown p;

- Test H0 : p = 0.6943 v.s. H1 : p > 0.6943 at α = 0.05.

- Possible rejection region: Reject H0 only if X = 3, i.e., wehave Pr(X = 3) = 0.05 = α.

- Hence, acceptance region is 0, 1, 2, 4, 5, 6, 7, 8Question: Is this a correct rejection region?

Solution: Yes, but it is not the best.

Consider three cases:1) Neyman-Pearson Lemma; 2) UMP; 3) GLR.

2117/2155


Rejection region



µ0

µ1 c x

f(x|H0) → ← f(x|H

1)

How toselect blueshaded area(rejectionregion) suchthat purplearea (i.e.,rejecting H0

given H1 istrue) is thelargest?Note: blueshaded areais Type Ierror = α.2118/2155


Rejection region


Formally: A subset C ? of the sample space is called a bestcritical region for testing the above hypothesis if for everysubset A of the sample space for which:

Pr ((X1,X2, . . . ,Xn) ∈ A |H0 ) = α,

the following two conditions hold:i) Pr ((X1,X2, . . . ,Xn) ∈ C? |H0 ) = α;ii) Pr ((X1,X2, . . . ,Xn) ∈ C? |H1 ) >

Pr ((X1,X2, . . . ,Xn) ∈ A |H1 ).

i) Implies that the probability of rejection null hypothesis, giventhat the null hypothesis is true, is equal to α;

ii) The probability that the H0 is rejected given H1 is true is thelargest for region C ?.

Used on slide 2120 for NP and on slide 2132 for UMP.

Optimal tests: tests with higher power than any other test,given a significance level α.2119/2155


Rejection region

Neyman-Pearson Lemma

Hypothesis testing





SummarySummary


Rejection region


Neyman-Pearson LemmaSuppose X1,X2, . . . ,Xn is a random sample from a distributionwhose density function is f (x ; θ). Let the joint density ofX1,X2, . . . ,Xn be:

L (x1, x2, . . . , xn; θ) = f (x1; θ)·f (x2; θ)·. . .·f (xn; θ) = Pr(X = x ; θ).

A subset C ? = x ; θ0, θ1 of the sample space is the best criticalregion for testing the simple null/simple alternative test statedabove, if the following conditions are satisfied (see slide 2119):

i) Pr ((X1,X2, . . . ,Xn) ∈ C ? |H0 ) = α;

ii) Λ(x ; θ0, θ1) =L (x1, . . . , xn; θ0)

L (x1, . . . , xn; θ1)6 k ,

for every (x1, . . . , xn) ∈ C ?, where k ≤ 1 is a constant suchthat: Pr ((X1, . . . ,Xn) ∈ C ?|θ0) = α.Hence: when rejecting the null, when it is incorrect (i.e.,(x1, . . . , xn) ∈ C ?) the likelihood of the data coming from thealternative is larger than from the null distribution.2120/2155


Rejection region


Proof Neyman-Pearson LemmaLet us denote C as the complement of set C ,X = [X1, . . . ,Xn] and x = [x1, . . . , xn] and A is ann-dimensional event, let:

Pr (X ∈ A|θ) =

∫

AfX (x ; θ) =

∫· · ·A

∫fX (x1, . . . , xn; θ)dx1 . . . dxn = L(x ; θ),

for any θ (continuous case).

Using condition ii) we obtain:- in case A ⊆ C?:

L(x ; θ0) ≤ k · L(x ; θ1) ⇒ Pr (X ∈ A|θ0)≤ k · Pr (X ∈ A|θ1)

- in case A ⊆ C?:

1− Pr(X ∈ C?|θ0) ≤k · (1− Pr(X ∈ C

?|θ1))

⇒ Pr(X ∈ C?|θ0) ≥k · Pr(X ∈ C

?|θ1) + (1− k) ≥ k · Pr(X ∈ C?|θ1)

⇒ Pr (X ∈ A|θ0)≥k · Pr (X ∈ A|θ1).2121/2155


Rejection region


For any critical region C we have (using LTP):

C ? = (C ? ∩ C ) ∪ (C ? ∩ C ) and C = (C ∩ C ?) ∪ (C ∩ C?).

Define for a set A and parameter θ: πA(θ) = Pr(X ∈ A|θ):

πC?(θ) =Pr(X ∈ C ? ∩ C |θ) + Pr(X ∈ C ? ∩ C |θ)

πC (θ) =Pr(X ∈ C ? ∩ C |θ) + Pr(X ∈ C ∩ C?|θ)

⇒ πC?(θ)− πC (θ) =Pr(X ∈ C ? ∩ C |θ)− Pr(X ∈ C ∩ C?|θ). (1)

* using (1) with θ = θ1 ** using previous slide *** using (1)with θ = θ0 **** using slide 2119 πA(θ0) = Pr (X ∈ A |H0 ) = α:

πC?(θ1)− πC (θ1)∗=Pr(X ∈ C ? ∩ C |θ)− Pr(X ∈ C ∩ C

?|θ)∗∗≥(1/k) ·

(Pr(X ∈ C ? ∩ C |θ0)− Pr(X ∈ C ∩ C

?|θ0))

∗∗∗= (1/k) · (πC?(θ0)− πC (θ0))

∗∗∗∗= (α− α)/k = 0

⇒ πC?(θ1) ≥πC (θ1).2122/2155


Rejection region


Interpretation Neyman-Pearson LemmaSuppose H0 specifies density or frequency function L (x ; θ0)and alternative H1 specifies L (x ; θ1);

Relative likelihoods of H0 versus H1 is the likelihood ratio:

L (x ; θ0) /L (x ; θ1) .

Question: What does a low ratio imply?

Solution: A low ratio implies that the data are more likelyunder H1 than under H0.

Thus: Likelihood ratio rejects for small values of the ratio.

Neyman-Pearson Lemma: if the likelihood ratio that rejectsH0 with significance level α is:

L (x ; θ0) /L (x ; θ1) < c ,

then any other test with significance level ≤ α will have powerless than or equal to that of the likelihood ratio test.2123/2155


Rejection region

Neyman-Pearson Lemma: examples and exercises

Hypothesis testing





SummarySummary


Rejection region


Example: Neyman-Pearson Lemma

Insurer AB is in the process of taking over local (and thussmall) insurer BC.

In the process of controlling the books of insurer BC theinsurer observes that the normally distributed claim sizes ofinsurer BC are larger than of insurer AB. This can be due to,for example:

- statistical uncertainty, due to small sample size of insurer BC;

- adverse selection effects.

Due to the LLN insurer AB knows that his claims areN(µAB , σ

2AB) distributed.

Insurer AB want to know whether the mean claims of insurerBC is x or µAB . Assume that µAB is a know constant and thevariances of both insurers are equal, known and equals σ2

AB .2124/2155


Rejection region


Example: Neyman-Pearson LemmaLet σ2 = σ2

AB , µ0 = x and µA = µAB . X1,X2, . . . ,Xn from anormal distribution with known variance σ2. Hypotheses:

H0 : µ = µ0 v.s. H1 : µ = µ1

with α a (given) constant.

Question: What is the best critical region?

Solution: Use Neyman-Pearson lemma.

1. The likelihood ratio is:

Λ(x ; θ0, θ1) =f0 (x)

f1 (x)=L(x ; θ0)

L(x ; θ1)=

∏ni=1

1σ√

2πexp

(−12σ2 (xi − µ0)2

)

∏ni=1

1σ√

2πexp

(−12σ2 (xi − µ1)2

)

=exp

(−12σ2

∑ni=1 (xi − µ0)2

)

exp(−12σ2

∑ni=1 (xi − µ1)2

) < k.

2125/2155


Rejection region


Question: when to reject H0?

Solution: reject for small values of the ratio L (x ; θ0) /L (x ; θ1).

Thus, by using a log-transformation (monotonic increasingtransformation) we have that we reject H0 when:

n∑

i=1

(xi − µ1)2 −n∑

i=1

(xi − µ0)2 < k1

(= log(k) · 2 · σ2

)

⇒n∑

i=1

(x2i −2xiµ1 + µ2

1

)−(x2i −2xiµ0 + µ2

0

)< k1

∗⇒ 2n · x (µ0 − µ1) + n(µ2

1 − µ20

)< k1

⇒n∑

i=1

xi · (µ0 − µ1) < k2

(= (k1−n

(µ2

1 − µ20

))/2)

is small, * usingn∑

i=1xi = n · x .

2126/2155


Rejection region


Therefore, we have:(a) if µ0 > µ1 ⇒ (µ0 − µ1) > 0 the likelihood ratio is small if x is

small;

(b) if µ0 < µ1 ⇒ (µ0 − µ1) < 0 the likelihood ratio is small if x islarge.

2. In the latter (former) case need to determine a value x0 suchthat Pr

(X > x0|H0

)= α (Pr

(X < x0|H0

)= α) if H0 is true.

We know: X |H0 ∼ N(µ0, σ

2/n)⇒ calculate the value of x0.

Pr(X < x0|H0

)= α ⇒ Pr

(X − µ0

σ/√n<

x0 − µ0

σ/√n

)= α

Pr (Z < −z1−α) = α ⇒ −z1−α =x0 − µ0

σ/√n

⇒ x0 = µ0 − z1−α · σ/√n.

Best critical region:

C ? =

(x1, . . . , xn) : x ≤ µ0 − z1−α · σ√n

in case of (a) and

C ? =

(x1, . . . , xn) : x ≥ µ0 + z1−α · σ√n

in case of (b).

2127/2155


Rejection region


Time until a natural disaster claims. Consider a random sample ofsize n with Xi ∼ EXP(θ). Test H0 : θ = θ0 v.s. H1 : θ = θ1 < θ0.

Exercise: Find the best critical region.

1. Solution: The Neyman-Pearson Lemma says to reject H0 if:

Λ(x ; θ0, θ1) =θn0 · exp (−θ0 ·

∑ni=1 xi )

θn1 · exp (−θ1 ·∑n

i=1 xi )≤ k ,

where k is a constant such that Pr(Λ(x ; θ0, θ1) ≤ k |H0) = α.

⇒n∑

i=1

xi · (θ1 − θ0) ≤ log((θ1/θ0)n · k) ⇒n∑

i=1

xi ≥ k1,

where k1 = (θ1 − θ0)−1

︸︷︷︸<0

· log((θ1/θ0)n · k).

Thus C ? = (x1, . . . , xn) :n∑

i=1xi ≥ k1.

Next step: Find k1 use Pr(X ∈ C?|θ0) = Pr(∑n

i=1 Xi ≥ k1|θ0

).

2128/2155


Rejection region


2. Using m.g.f. technique (see week 4) we have:∑ni=1 Xi |H0 ∼ Gamma(n, θ0)⇒ 2θ0

∑ni=1 Xi |H0 ∼ χ2(2n),

Pr

(n∑

i=1

Xi ≥ k1|H0

)= α ⇒ Pr (2 · θ0

n∑

i=1

Xi

︸︷︷︸∼χ2(2n)

≥ 2 · θ0 · k1︸︷︷︸χ2

1−α(2n)

|H0) = α

⇒ χ21−α(2n) = 2 · θ0 · k1 ⇒ k1 =

χ21−α(2n)

2 · θ0.

Thus: C ? =

(x1, . . . , xn) :

n∑i=1

xi≥ χ21−α(2n)/(2 · θ0)

.

Exercise: Consider a random sample of size n, with0 ≤ xi ≤ 1 for i = 1, . . . , n.Test H0 : X ∼ UNIF(0, 1) with fX (xi ) = 1 v.s.H1 : X ∼ EXP(1) with fX (xi ) = e−xi .Note: under H0: E[Xi ] = 0.5 and Var(Xi ) = 1/12.

2129/2155


Rejection region


1. Solution: The Neyman-Pearson Lemma says to reject H0 if:

Λ(x ; θ0, θ1) =1

exp (−∑ni=1 xi )

≤ k ,

where k is a constant such that Pr(Λ(x ; θ0, θ1) ≤ k |H0) = α.So we reject H0 if

∑ni=1 xi ≤ k1 = log(k).

2. How to determine the distribution of∑n

i=1 Xi |H0, use CLT:

Zn =

∑ni=1 Xi − E[

∑ni=1 Xi ]√

Var(∑n

i=1 Xi )=

∑ni=1 Xi − n · E[Xi ]√

n · Var(Xi )

=

∑ni=1 Xi − n/2√

n/12

d→ Z ∼ N(0, 1)

⇒ Pr (Zn < −z1−α) = α ⇒ Pr

(n∑

i=1

Xi ≤n

2− z1−α ·

√n/12

)= α

Thus C ? =

(x1, . . . , xn) :

n∑i=1

xi≤ n2 −

z1−α·√n√

12

.

2130/2155


Rejection region

Uniformly most powerful (UMP)

Hypothesis testing





SummarySummary


Rejection region


Uniformly most powerful

Consider simple null, composite alternative hypothesis, i.e.,H0 : θ = θ0 v.s. H1 : θ ∈ ω1.

Question: When applying Neyman-Pearson Lemma whatwould be L (x1, x2, . . . , xn; θ1)?

Answer: This would depend on the θ1 ∈ ω1 which is true.

The critical region C is uniformly most powerful (UMP) ofsize α for testing a simple H0 against a composite H1 if C isthe best critical region of size α for testing H0 againstevery simple hypothesis in H1. The corresponding test iscalled the UMP test of size α.

If H1 composite then a uniformly most powerful test is mostpowerful for every simple alternative in H1.

2131/2155


Rejection region


Uniformly most powerful

Let X1, . . . ,Xn have joint p.d.f. f (x1, . . . , xn; θ) for θ ∈ Ω.Consider hypothesis of the form:

H0 : θ ∈ Ω0 v.s. H1 : θ ∈ Ω− Ω0.

A critical region C ? and the associated test are said to beuniformly most powerful (UMP) of size α if the following twoconditions hold (using slide 2119):

i) maxθ∈Ω0

Pr((X1, . . . ,Xn) ∈ C?|θ) = α;

ii) for all θ ∈ Ω− Ω0 and all critical regions C of size α we have:

Pr((X1, . . . ,Xn) ∈ C?|θ) ≥ Pr((X1, . . . ,Xn) ∈ C |θ)

i.e., the probability of rejecting the null, when the null isincorrect (i.e., θ ∈ Ω− Ω0) is the largest for critical region C?.

2132/2155


Rejection region


A joint p.d.f. f (x ; θ) is said to have a monotone likelihoodratio (MLR) in the statistic T = τ(X ) if for any two values ofthe parameter θ1 < θ2, the ratio f (x ; θ2)/f (x ; θ1) depends onX only through the function τ(x), and this ratio is anondecreasing function of τ(x).

Neyman-Pearson tests also provide UMP test forcorresponding one-sided composite alternatives.

If a joint p.d.f. f (x ; θ) has a MLR in T = τ(X ) then a UMPtest of size α for H0 : θ ≤ θ0 v.s. H1 : θ > θ0 is to reject H0 ifτ(x) ≥ k , where Pr(τ(X ) ≥ k|θ0) = α.

Example: Consider the example from slide 2128.We have τ(X ) =

∑ni=1 Xi and

Λ(x ; θ0, θ1) =θn0 · exp (−∑n

i=1 xi · θ0)

θn1 · exp (−∑ni=1 xi · θ1)

=

(θ0

θ1

)n

· exp

(n∑

i=1

xi · (θ0 − θ0)

)

is thus a nondecreasing function in τ(x) if θ1 < θ0.2133/2155


Rejection region

Example: Uniformly most powerful (UMP)

Hypothesis testing





SummarySummary


Rejection region


Example UMP: testing means from normal distribution

Consider sampling from the normal distribution to test themean parameter.

Same example, but now test whether the claims sizes ofinsurer BC are higher.

Suppose X1,X2, . . . ,Xn is a random sample from N(µ, σ2

).

We wish to test:

H0 : µ = µ0 v.s. H1 : µ > µ0,

for some known constant µ0 and α.

If the variance σ2 is known, we know from week 5:

X ∼ N(µ, σ2/n).2134/2155


Rejection region


Example UMP: testing means from normal distribution1. The Likelihood ratio numerator f0(x) = L(x ;µ0) is constant:

f0(x) =

(1√2πσ

)n

· exp

(−1

2σ2

n∑

i=1

(xi − µ0)2

).

The Likelihood ratio denominator f1(x) = L(x ; θ1) depends onµ1 > µ0:

f1(x) =

(1√2πσ

)n

· exp

(−1

2σ2

n∑

i=1

(xi − µ1)2

).

for all µ1 > µ0.

We have:

Λ(x ;µ0, µ1) =exp

(−12σ2

∑ni=1 (xi − µ0)2

)

exp(−12σ2

∑ni=1 (xi − µ1)2

) < k ,

for all µ1 > µ0.

2135/2155


Rejection region


Reject H0 for small values of f0(x)/f1(x), i.e., (using alog-transformation and simplifying) when:

x · (µ0 − µ1) < k1

is small (for all µ1 > µ0).

We have: µ0 − µ1 < 0 for all µ1 > µ0, thus:

x ≥ k?(

=k1

(µ0 − µ1)

),

2. Next, determine a value x0 such that Pr(X > k?|H0

)= α.

We know that X |H0 ∼ N(µ0,

σ2

n

)⇒ calculate the value of

k? = µ0 + z1−α · σ/√n.

Best critical region:

C ? =

(x1, . . . , xn) : x ≥ µ0 + z1−α · σ√n

.

2136/2155


Rejection region


Exercise UMP

Number of claim distribution. Consider a sample of size nfrom a Poisson distribution. Xi ∼ Poi(λ), with joint p.d.f.:

fX (x ;λ) =exp(−nλ) · λ

∑ni=1 xi

x1! · . . . · xn!

=(x1! · . . . · xn!)−1 · exp

(log(λ) ·

n∑

i=1

xi − n · λ)

Exercise: use MLR to find the UMP of size α for H0 : λ ≤ λ0

v.s. H1 : λ > λ0.

1. Solution: We have:

f (x ;λ1)

f (x ;λ0)= exp

((log(λ1)− log(λ0)) ·

n∑

i=1

xi − n · (λ1 − λ0)

)

2137/2155


Rejection region


(continued) τ(λ) =∑n

i=1 xi , reject H0 if T =∑n

i=1 Xi ≥ k,where Pr(T ≥ k|λ0) = α.

2. We have T =∑n

i=1 Xi ∼ POI(nλ0), thus

k2 = argmaxk1

∞∑

x=k1

exp (−nλ0) · (nλ0)x

x!≤ α

Rejection region: C ? = (x1, . . . , xn) |∑ni=1 xi ≥ k2 .

Dental insurance: Consider a sample of size n from a two-parameter exponential distr, Xi ∼ EXP(1, η). The j.p.d.f. is:

fX (x , η) =

exp (−∑n

i=1(xi − η)) , if η < x(1);0, if x(1) ≤ η.

Example: use MLR to find the UMP of size α for H0 : η ≤ η0

v.s. H1 : η > η0.2138/2155


Rejection region


Distribution of minimum:Pr(X(1) ≥ k) = Pr(X1 ≥ k , . . . ,Xn ≥ k) = Pr(Xi ≥ k)n.

1. Solution: we have:

f (x , η1)

f (x , η0)=

0, if η0 < x(1) ≤ η1;exp (n · (η1 − η0)), if η1 < x(1).

Note that f (x , η1)/f (x , η0) is not defined for x(1) ≤ η0, butthis is no problem because Pr(X(1) ≤ η0|η0) = 0.

T = τ(η) = X(1), f (x , η1)/f (x , η0) is nondecreasing functionin x(1), hence we can use MLR.

2. Find rejection region by:

α = Pr(T ≥ k |η0) = Pr(X(1) ≥ k |η0) = exp(−n · (k − η0))

⇒ k =η0 − log(α)/n.

Rejection region: C ? =

(x1, . . . , xn) : x(1) ≥ η0 − log(α)/n

.2139/2155


Rejection region

Generalized likelihood ratio test

Hypothesis testing





SummarySummary


Rejection region


Generalized likelihood ratio testSummarizing the determination of the critical region we have:

- Neyman-Pearson: when simple null v.s. simple alternative;

- UMP: in case of simple null v.s. composite alternative(example H1 : µ > µ0, but similarly we have H1 : µ < µ0);

- Can we use UMP for simple null v.s. composite alternativewith µ1 6= µ0?No, C = (x0,∞) in case µ > µ0, but C = (−∞, x0) in caseµ < µ0.

When there is no uniformly most powerful test for testing asimple null against a composite alternative hypothesis. In thiscase the generalized likelihood ratio test (LRT) is used.

The generalized LRT is also the basis for constructing a test ofa composite null against a composite alternative hypothesis.

Note: LRT tests do not always lead to the best critical region.2140/2155


Rejection region


Best critical region, 2-sided tests

µ0

µ1

µ0 + z1−α/2σ√n

µ0 − z1−α/2σ√n

f(x|H0) → ← f(x|H

1)

How toselect blueshaded area(rejectionregion) suchthat purplearea (i.e.,rejecting H0

given H1 istrue) is thelargest?Note: blueshaded areais Type Ierror = α.2141/2155


Rejection region



Suppose X1,X2, . . . ,Xn is a random sample from f (x ; θ) sothat the joint density is:

L (x1, x2, . . . , xn; θ) = f (x1; θ) · f (x2; θ) · . . . · f (xn; θ) .

Denote the test by:

H0 : θ ∈ Ω0 v.s. H1 : θ ∈ Ω1, for a given α,

where Ω = Ω0 ∪ Ω1.

Define the likelihood functions:

LΩ0 (θ) = L (x1, x2, . . . , xn; θ) , with θ ∈ Ω0,LΩ (θ) = L (x1, x2, . . . , xn; θ) , with θ ∈ Ω.

2142/2155


Rejection region


Denote their maximums, if they exist, by maxθ∈Ω0

LΩ0 (θ) and

maxθ∈ΩLΩ (θ), respectively.

Denote θ0 the MLE under the restriction that H0 is true and θunder the restriction that H0 or H1 is true (typically the usualMLE).

The ratio:

Λ (x1, x2, . . . , xn) =

maxθ∈Ω0

LΩ0 (θ)

maxθ∈ΩLΩ (θ) =

L(x ; θ0)

L(x ; θ),

is called the likelihood ratio. Note: 0 ≤ Λ (x) ≤ 1.

The LRT principle says that the null hypothesis above isrejected if and only if:

Λ (x1, x2, . . . , xn) < k ,

for some positive k.2143/2155


Rejection region

Asymptotic distribution for the GLR test

Hypothesis testing





SummarySummary


Rejection region


GLR is similar to Neyman-Pearson principle.

GLR is not a complete generalization of Neyman-PearsonLemma, because the unrestricted estimate θ could be possiblein Ω0.

The function Λ (X ) is a valid test statistic ⇒ not a functionof unknown parameters:

- Λ (X ) is free of parameters: exact critical value k can bedetermined.

- Λ (X ) under H0 depends on unknown parameters ⇒ MLE areasymptotically normally distributed (under regulatoryconditions) ⇒ free of parameters. If X ∼ f (x ; θ1, . . . θk) thenunder H0 : (θ1, . . . , θr ) = (θ10, . . . , θr0), r < k we haveapproximately for large n:

−2 log(Λ(X )) ∼χ2(r)

An approximate size α test to reject H0 if:

−2 log(Λ(X )) ≥χ21−α(r)2144/2155


Rejection region


Asymptotic Distribution of Likelihood Ratio

Recall from last week asymptotic MLE property:

√nIf ?(θ)(θ − θ)

d→ N(0, 1).

Under certain “smoothness” condition of the probabilitydensity (or mass) functions involved, the distribution of thetest statistic:

T (X ) = −2 log (Λ) (= −2 (log (L(x ; θ0))− log (L(x ; θ))))

has -under the null hypothesis- asymptotically a chi-squareddistribution with degrees of freedom dim (Ω)− dim (Ω0).

This asymptotic result often used for the (approximate)distribution of the test statistic under the null hypothesis.

2145/2155


Rejection region

Example: generalized likelihood ratio test

Hypothesis testing





SummarySummary


Rejection region


Example LRT: testing means from normal distributionConsider sampling from the normal distribution to test themean parameter.

Suppose X1,X2, . . . ,Xn is a random sample from N(µ, σ2

).

We wish to test:

H0 : µ = µ0 v.s. H1 : µ 6= µ0,


1. Define the sets:

Ω0 =µ = µ0 and Ω1 = µ ∈ (−∞, µ0) ∪ (µ0,∞)thus Ω =(−∞,∞).

2. Find the θ ∈ Ω and θ ∈ Ω0 that maximizes the likelihoodfunction (see week 5 MLE):

θ0 = µ0 and θ = x .2146/2155


Rejection region


Example LRT: testing means from normal distribution

3. Thus the Likelihood ratio numerator is:

maxθ∈Ω0

LΩ0(θ0) =

(1√2πσ

)n

· exp

(−1

2σ2

n∑

i=1

(xi − µ0)2

).

and the likelihood denumerator is:

maxθ∈ΩLΩ(θ) =

(1√2πσ

)n

· exp

(−1

2σ2

n∑

i=1

(xi − x)2

).

Next slide: the likelihood ratio(using exp(a)/ exp(b) = exp(a− b));and ** using nx − 2x

∑ni=1 xi = nx − 2nx2 = −nx .

2147/2155


Rejection region


Λ(x) = exp

(−1

2σ2

n∑

i=1

((xi − µ0)2 − (xi − x)2

))< k

Rejecting for small values of Λ is the same as rejecting for largevalues of:

⇒n∑

i=1

((xi − µ0)2 − (xi − x)2

)> k1 (= −2σ2 log (k)︸︷︷︸

decreasing

)

⇒n∑

i=1

x2i + nµ2

0 − 2µ0

n∑

i=1

xi −(

n∑

i=1

x2i + nx2 − 2x

n∑

i=1

xi

)

∗∗= n · (x − µ0)2 > k1

⇒ (x − µ0)2 > k2 (= k1/n)

⇒ x − µ0 > |k3|(

= ±√k2

)

equivalently: µ0 − k? > x > µ0 + k?.2148/2155


Rejection region


Example LRT: testing means from normal distributionThe rejection region is of the form:

C ? = (x1, . . . , xn) : x ∈ (−∞, µ0 − k?) ∪ (µ0 + k?,∞) .where k? is such that:

Pr(µ0 − k? ≥ X ≥ µ0 + k?

∣∣H0

)= α.

4. If the variance σ2 is known, we know from week 5 (or week 4m.g.f. technique): X |H0 ∼ N(µ0, σ

2/n), thus:

Pr

( −k?σ/√n≥ X − µ0

σ/√n≥ k?

σ/√n

∣∣∣∣H0

)= α

Pr(−z1−α/2 ≥ Z ≥ z1−α/2

)= α ⇒ k? = z1−α/2 · σ/

√n.

Hence, the rejection region is:

C? =

(x1, . . . , xn) : x ∈

(−∞, µ0 −

σ√nz1−α/2

)∪(µ0 +

σ√nz1−α/2,∞

).

2149/2155


Rejection region


Example LRT: testing means from normal distributionAgain, suppose X1,X2, . . . ,Xn is a random sample fromN(µ, σ2

), and x > µ0. We wish to test:

H0 : µ ≤ µ0 v.s. H1 : µ > µ0,


What is then the Likelihood ratio numerator?

Given ∂L(µ, σ)/∂µ < 0 if µ > x , ∂L(µ, σ)/∂µ = 0 if µ = x ,and ∂L(µ, σ)/∂µ > 0 if µ < x we have:

µ =

x , if x < µ0, i.e., the MLE;µ0, if x ≥ µ0, i.e., bounding condition,

where µ is the µ ∈ Ω0 which maximizes LΩ0(µ).

Note: the Likelihood ratio denominator is the same as in theprevious example.2150/2155


Rejection region


Exercise LRTTesting equality in proportions of insured issuing claims.

Let X ∼ Bin(n1, p1) and Y ∼ Bin(n2, p2) with X and Yindependent.

Exercise: Test whether the proportions are equal, i.e.,H0 : p1 = p2 = p v.s. H1 : p1 6= p2, where p is unknown.

Solution: Based on x and y the MLEs are p1 = x/n1 andp2 = y/n2 and the restricted MLE is p = (x + y)/(n1 + n2).

The GLR statistic is:

Λ(x , y) =f (x ; p) · f (y ; p)

f (x ; p1) · f (y ; p2)=

(n1x

)· px · (1− p)n1−x ·

(n2y

)· py · (1− p)n2−y

(n1x

)· px1 · (1− p1)n1−x ·

(n2y

)· py2 · (1− p2)n2−y

=px+y · (1− p)n1+n2−x−y

px1 · (1− p1)n1−x · py2 · (1− p2)n2−y2151/2155


Rejection region


Exercise LRT

GLR test statistic under H0 depends on unknown parameter p.

Note rewrite H0 : θ = p2 − p1 = 0 so H0 represents aone-dimensional restriction in the parameters, i.e., r = 1 freeparameter.

An approximately size α test is:

−2 log(Λ(x , y)) ∼ χ2(1)

Hence, H0 is rejected if −2 · log(Λ(x , y)) > χ21−α(1), i.e.,

C ? =

(x1, . . . , xn) : −2 · log(Λ(x , y)) > χ21−α(1)

2152/2155


Summary

Summary

Hypothesis testing





SummarySummary


Summary

Summary

Hypothesis tests

Testing procedure: When testing a hypothesis use the followingsteps:

i. Define a statistical hypothesis.Note that this includes a confidence level (α);

ii. Define the test statistic T (using past weeks knowledge);

iii. Determine the rejection region C ?;

iv. Calculate the value of the statistical test, given observed data(x1, . . . , xn);

v. Accept or reject H0.Note: we assume that H0 is true when testing! (see Type Iand Type II errors)

2153/2155


Summary

Summary

Best critical regionNeyman-Pearson Lemma: simple null/simple alternativetest:

Λ(x ; θ0, θ1) =L(x ; θ0)

L(x ; θ1)< k s.t.: Pr (X ∈ C ?|H0) = α

Uniform most powerful test: simple/compositenull/composite alternative test:

i) maxθ∈Ω0

Pr((X1, . . . ,Xn) ∈ C?|θ) = α;

ii) for all θ ∈ Ω− Ω0 and all critical regions C of size α we have:

Pr((X1, . . . ,Xn) ∈ C?|θ) ≥ Pr((X1, . . . ,Xn) ∈ C |θ)

Generalized likelihood ratio test: simple/compositenull/composite alternative test:

Λ(x ; θ0, θ) =L(x ; θ0)

L(x ; θ)< k s.t.: Pr (X ∈ C ?|H0) = α.

2154/2155


Summary

Summary

2155/2155

Week 7 Annotated

Documents

Transcript of Week 7 Annotated