Why you need power analysis
-
Upload
pcdjohnson -
Category
Science
-
view
470 -
download
0
Transcript of Why you need power analysis
![Page 1: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/1.jpg)
Why you need power analysis even if you think you don’t Paul Johnson1 Sarah Barry2 Heather Ferguson1 Pie Müller3
1: Boyd Orr Centre, IBAHCM, University of Glasgow
2: Robertson Centre for Biostatistics, University of Glasgow
3: Swiss Tropical and Public Health Institute
14/12/2015 @paulcdjo 1
![Page 2: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/2.jpg)
Power
Narrow definition: The probability of a significant result given some effect size
• Applies only to null hypothesis significance testing (NHST)
Broad definition: The information we expect to gain from a study that uses statistical inference; “informativeness”
• Applies to all modes of statistical inference:
• Null hypothesis significance testing (NHST)
• Estimating an effect & confidence interval
• Information theoretic (AIC, BIC, etc)
• Bayesian
14/12/2015 @paulcdjo 2
![Page 3: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/3.jpg)
Reproducibility
“Reproducibility is the ability of an entire experiment or study to be duplicated… Reproducibility is one of the main principles of the scientific method.”
Wikipedia, 08/12/15
“non-reproducible single occurrences are of no significance to science”
Karl Popper, The Logic of Scientific Discovery, 1935
14/12/2015 @paulcdjo 3
![Page 4: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/4.jpg)
Low power means low reproducibility
• 1000 studies test a null hypothesis (H0) at 5% significance level and with 30% power
• In 100, H0 is false, 30% x 100 = 30 significant and true
• In 900, H0 is true, 5% x 900 = 45 significant but false
• P(true | significant) = 30/(30 + 45) = 40%
14/12/2015 @paulcdjo 4
![Page 5: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/5.jpg)
A crisis of irreproducibility?
14/12/2015 @paulcdjo 5
![Page 6: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/6.jpg)
A crisis of irreproducibility?
• Human genetics
• Psychology
• Neuroscience
14/12/2015 @paulcdjo 6
![Page 7: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/7.jpg)
Irreproducibility in human genetics
14/12/2015 @paulcdjo 7
Nature Genetics 29, 306-309 (2001)
Typical GWAS sample sizes from current Nature Genetics issue:
• 23,000
• 27,000
• 116,000
![Page 8: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/8.jpg)
Irreproducibility in psychology
14/12/2015 @paulcdjo 8
Original study effect size versus replication effect size
(correlation coefficients).
Open Science Collaboration
Science 2015;349:aac4716
Reproducibility Project: Psychology
• Aimed to replicate 100 studies
• Result: 39/100 replicated original result
![Page 9: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/9.jpg)
Irreproducibility in neuroscience
14/12/2015 @paulcdjo 9
Nature Reviews Neuroscience 14, 365-376 (2013)
“Our results indicate that the median statistical power in neuroscience is 21%.” “The consequences of this [very low power] include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.”
![Page 10: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/10.jpg)
14/12/2015 @paulcdjo 10
![Page 11: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/11.jpg)
A crisis of irreproducibility?
• Human genetics
• Psychology
• Neuroscience
14/12/2015 @paulcdjo 11
![Page 12: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/12.jpg)
A crisis of irreproducibility?
• Human genetics
• Psychology
• Neuroscience
• Ecology?
14/12/2015 @paulcdjo 12
![Page 13: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/13.jpg)
A crisis of irreproducibility in ecology?
• No study has directly assessed reproducibility across ecology
• Are known causes of irreproducibility prevalent in ecological research?
14/12/2015 @paulcdjo 13
![Page 14: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/14.jpg)
Low power in ecological research?
• Jennions & Møller (2003) A survey of the statistical power of research in behavioral ecology and animal behavior. Behavioral Ecology • Average power 40-47% for medium effect sizes
• Taborsky (2010) Sample size in the study of behaviour. Ethology • Power analysis rarely used in behavioural research
• Smith et al. (2011) Power rangers: no improvement in the statistical power of analyses published in Animal Behaviour. Animal Behaviour • Average power 23-26% for medium effect sizes
What can we do about it?
14/12/2015 @paulcdjo 14
![Page 15: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/15.jpg)
14/12/2015 @paulcdjo 15
Obstacle to using power analysis Solution
Power analysis doesn’t apply to my study
It does if you are using statistical inference – broaden your definition of power analysis
I don’t know what parameter values to assume (pilot data not available)
Assume a plausible worst-case scenario based on experience, or do a pilot study
I don’t know the true effect size You don’t need to – this is the job of the study. Think instead of • The smallest effect worth detecting (for NHST) • The desired precision of the estimate (for estimation)
My model is too complicated for power analysis
• Simplify to the bare essentials • Use simulations • Ask for help
Power analysis is hard & no one is forcing me to do it
Power analysis should be promoted by • Journal editors • Funders • Group leaders
![Page 16: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/16.jpg)
14/12/2015 @paulcdjo 16
![Page 17: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/17.jpg)
14/12/2015 @paulcdjo 17
![Page 18: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/18.jpg)
14/12/2015 @paulcdjo 18
![Page 19: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/19.jpg)
14/12/2015 @paulcdjo 19
Obstacle to using power analysis Solution
Power analysis doesn’t apply to my study
It does if you are using statistical inference – broaden your definition of power analysis
I don’t know what parameter values to assume (pilot data not available)
Assume a plausible worst-case scenario based on experience, or do a pilot study
I don’t know the true effect size You don’t need to – this is the job of the study. Think instead of • The smallest effect worth detecting (for NHST) • The desired precision of the estimate (for estimation)
My model is too complicated for power analysis
• Simplify to the bare essentials • Use simulations • Ask for help
Power analysis is hard & no one is forcing me to do it
Power analysis should be strongly encouraged by • Journal editors • Funders • Group leaders
![Page 20: Why you need power analysis](https://reader031.fdocuments.us/reader031/viewer/2022022413/58ee06901a28ab3c418b45d1/html5/thumbnails/20.jpg)
Conclusions
• Underpowered studies are probably common in ecology, so irreproducibility is probably rife
• Time and money spend on underpowered research is wasted
• We should stop looking for excuses to avoid power analysis
• We should collaborate with statisticians who know how to do power analysis
• Journal editors, funders & group leaders should encourage power analysis
Funding acknowledgement: AvecNet - African Vector Control: New Tools
14/12/2015 @paulcdjo 20