Sample Size And PowerWarren Browner and Stephen Hulley

The ingredients for sample size planning, and how to design them

An example, with strategies for minimizing sample size

Sampling and Inference

A sample is designed to represent a larger population

Therefore, findings in the sample allow inferences about events in the population

Problem: what if the inferences are wrong? Finding something in the sample that isn’t “real”

in the population Missing something that is “real”

Preventing Wrong Inferences

Difficult when caused by systematic error (bias)

Easier when caused by Random error (chance) Solution: increase sample size Problem: cost, feasibility Goldilocks solution: a sample size that

is big enough but not too big

Ingredients For Planning Sample Size in an Analytic Study or RCT

Hypothesis Null and alternative One-sided and two-sided

Statistical test Type of variables

Effect size (and its variance) Power and alpha

Research Hypothesis

A clear statement of what you are studying.

Simple: one predictor, one outcome

Specific: who, what, when, where

Stated: in advance

Research Hypothesis

In patients with early ALS seen at UCSF in 2007, those randomly assigned to be treated with minocycline will have a lower 1-year mortality than those randomly assigned to placebo.

The Null Hypothesis

There’s nothing going on.

Purpose in life: to be rejected in favor of its alternative.

In patients with early ALS seen at UCSF in 2007, those randomly assigned to be treated with minocycline will have the same 1-year mortality as those randomly assigned to placebo.

What’s This All About?

A long time ago, statisticians figured out the probability that a sample of a given size would “find something” even if there were nothing going on in the population.

This means that...

After a study, we can determine the likelihood that whatever we found in our sample could have occurred by chance... Even if nothing was going on in the

population (i.e., the null hypothesis was true)--a “Type I error”

If this is very unlikely (say < 1 in 20) we reject the null hypothesis in favor of the alternative hypothesis; we call the finding statistically significant (P < .05)

Two-sided Alternative Hypothesis

In patients with early ALS seen at UCSF in 2007, those randomly assigned to be treated with minocycline will have a different 1-year mortality than those randomly assigned to placebo.

Two One-sided Alternative Hypotheses

Side A: In patients with early ALS seen at UCSF in 2007, those randomly assigned to be treated with minocycline will have a higher 1-year mortality than those randomly assigned to placebo.

Side B: In patients with early ALS seen at UCSF in 2007, those randomly assigned to be treated with minocycline will have a lower 1-year mortality than those randomly assigned to placebo.

If The Null Hypothesis Is True

By chance alone, each of the two one-sided alternative hypotheses is... Possible Wrong Equally likely

Thus a two-sided alternative hypothesis has twice the likelihood of happening by chance alone

Next Ingredient: Statistical Test (Types of Variable)

The statistical test determines how the sample size will be calculated

The type of predictor and outcome variable determine which statistical test will be used to analyze the data Both dichotomous: Chi square One dichotomous, one “continuous”: t test Both “continuous”: correlation coeff or t test

Statistical Test (Types of Variable)

ALS study Predictor: minocycline vs placebo Outcome: % dead

Both are dichotomous Chi square test

Next Ingredient:Effect Sizes (dichotomous variables)

How big an effect you anticipate seeing

Minocycline halves mortality

Minocycline = 5%, Placebo = 10%

Penultimate Ingredient: Power

The chance of finding something in your sample if it’s really going on in the population (avoiding a Type II error) “Something” = the effect size (or greater)

Usually set at 80% or 90% = (1 - beta)

…and the Final Ingredient: Alpha

The chance of finding something in your sample if there’s nothing going on in the population.

Alpha Explained

The level of statistical significance (ie, the p-value that will be considered significant)

The pre-set maximum chance of finding something, if it really isn’t there.

Usually set at 0.05.

May be one-sided or two-sided.

Sidedness Of Alpha

With a two-sided alternative hypothesis, you have two chances of finding something that isn’t really there: One (equal) chance for each side.

So a one-sided alpha of 0.05 corresponds to a two-sided alpha of 0.10.

SAMPLE SIZE: AN EXAMPLE

Null hypothesis: In patients with early ALS seen at UCSF in

2007, those randomly assigned to be treated with minocycline will have the same 1-year mortality as those randomly assigned to placebo.

Two-sided alternative hypothesis Dichotomous predictor and outcome Effect size: 10% mortality + 5% Power, alpha: 90%, 0.05 (two-sided)

THE SAMPLE SIZE IS…

Appendix 6.B Smaller of P1 and P2 = 0.05; power of

90%; alpha of 0.05 (two-sided) Difference = 0.05

381473620

This is per group

Sample Size Reduction Strategy #1:Statistical Manipulation

Use a lower power Use a one-sided alpha

Power of 80% One-sided alpha of 0.05

The New Sample Size Is…

Appendix 6.B Smaller of P1 and P2 = 0.05; power of

80%; alpha of 0.05 (one-sided) Difference = 0.05

381473620

This is also per group

SS Reduction Strategy #2: Use A More Common Outcome

Change from 1-year mortality to 2-year mortality or loss of independent living

Placebo: 40% Minocycline: 20%

The New Sample Size Is…

Appendix 6.B Smaller of P1 and P2 = 0.20; power

of 80%; alpha of 0.05 (two-sided) Difference = 0.20

7491

118

SS Reduction Strategy #3: Use A Continuous Outcome

Change “mortality or loss of independent living” to “muscle strength”

NOTE: Big change in research question and research hypothesis.

New null hypothesis: In patients with early ALS seen at UCSF in

2007, those randomly assigned to be treated with minocycline will have the same grip strength at the end of six months as those treated with placebo.

Two-sided alternative hypothesis

Estimate The Mean And Variability Of Grip Strength

Patients with untreated ALS have a (mean ± SD) grip strength of 20 ± 10 kg after 6 months of disease

Minocycline may improve that by 25%

Then, at End of Study

Grip strength Placebo: 20 kg Minocycline: 25 kg (25% more)

Effect size = 5 kg SD = 10 kg

Standardized effect size: E/S = 5/10 = 0.5

The New Sample Size Is ...

Appendix 6.A E/S = 0.5 ß = 0.20, Alpha (two-sided) = 0.05

N = 64 per group

Ss Reduction Strategy #4: Use A More Precise Outcome

Buy a better instrument to measure grip strength

Use a well-defined protocol Repeat measurements on two

consecutive days

Reduce SD from 10 kg to 8 kg

The New Sample Size Is ...

New E/S = 5 kg/8 kg= 0.625 ß = 0.20, Alpha (two-sided) = 0.05

N = about 45 per group

This helped quite a bit.

SS Reduction Strategy #5: Use Paired Measurements

Most of the variability in grip strength at the end of the study is likely to be due to differences between subjects in grip strength at the beginning of the study.

Switch the outcome to change in grip strength from the beginning to the end of the study.

Paired Measurements

Each subject contributes a pair of measurements: (before, after)

The outcome variable is the difference between that pair for each subject.

The SD of the change in a measurement is usually < than the SD of the measurement

SD of change in grip strength is 5 kg New standardized effect size = 5/5 = 1.0

The New Sample Size Is...

E/S = 1.0 ß = 0.20, Alpha (two-sided) = 0.05

N = 17 per group

We now have a potentially do-able study, albeit one that is very different from the original aim.

The Bottom Line

Sample size estimation is an integral part of study planning

Almost never the last thing you do

More often, one of your first tasks

SAMPLE SIZE PLANNING: REVIEW OF INGREDIENTS

Looking for something in a sample Hypotheses (null and alternative) Will you be able to ...

Know it’s there in the population if you find it in your sample (avoid a Type I error)

Test of significance, alpha Find it in your sample if it’s there in the

population (avoid a type II error)? Effect size, power