STAT 3130Statistical Methods II

Session 6Statistical Power

STAT3130 – Statistical Power

Type 1 Error (alpha)

Power (1-beta)

1-alpha Type 2 Error (beta)

Ho TRUE Ho FALSE

Reject Ho

FTR Ho

When we engage in Hypothesis Testing, we have focused on the p-value and the alpha value…recall what these actually mean.

STAT3130 – Statistical Power

Type 1 Error (alpha)

Power (1-beta)

1-alpha Type 2 Error (beta)

Ho TRUE Ho FALSE

Reject Ho

FTR Ho

A second calculation of interest is the Power of the test – or 1- the probability of committing a Type 2 error. In words, the Power of the test is “The probability of correctly rejecting the Null Hypothesis when it should be rejected”.

In other words, it is the probability that if a true difference exists, it will be discovered. It is sometimes referred to as the “sensitivity” of the test.

Statistical Power is heavily used in medicine, clinical psychology and biology.

Typically, a test must have a Statistical Power of 80% or greater to be considered valid. Why would that be?

STAT3130 – Statistical Power

Power is a function of three factors:

1. Effect size – i.e., the difference between the two groups or measurements. As the effect size goes up, the power increases.

2. Alpha – As the chance of finding an incorrect significant effect is reduced (Type I error), the probability of correctly finding an effect is also reduced. Typically, alpha is set to be .01 (most conservative and lowers power), .05 or .10 (most risk tolerant and increases power). As alpha goes down, the power increases.

3. Sample Size - Increased sample sizes will always produce greater power. But, increasing the sample size can also produce “too much” power – smaller and smaller effects will be found to be significant until at a large enough sample size, any effect is considered to be significant. As sample size goes up, power increases.

STAT3130 – Statistical Power

When conducting a one tailed test, the power calculation is executed as:

1-() = 1-P[z<z- (| 0 - a|/ (s/SQRT(n))]

A two tailed test is conducted similarly, with the alpha value associated with the z score divided by two.

STAT3130 – Statistical Power

Example…

Lets say that we have a treatment group taking a cholesterol lowering drug. Lets say that the average decrease in cholesterol after taking the drug is 2 points. (n=100 and s = 10). What do you conclude.

1-(2) = 1-P[z<1.645- (| 2|/ (10/SQRT(100))]

Now…what if the drop is 3 points. What happens?

STAT3130 – Statistical Power

Lets build some power curves…

Power versus effect size

Power versus sample size

STAT3130 – Statistical Power

If you are starting an experiment from scratch, how do you determine the appropriate sample size?

n= 2*(Z +Z )2/Δ2

A two tailed test is conducted similarly, with the alpha value associated with the z score divided by two.

(Or…you can just use SAS).

STAT3130 – Statistical Power