P-Hacking: A Practical Guide

@Neuro_Skeptic

neuroskeptic@googlemail.comhttp://blogs.discovermagazine.com/neuroskeptic

The Nine Circles of Dante’s Inferno

First Circle: Limbo

Second Circle: Lust

Third Circle: Gluttony

Fourth Circle: Greed

Fifth Circle: Anger

Sixth Circle: Heresy

Seventh Circle: Violence

Eighth Circle: Fraud

Ninth Circle: Treachery

The Nine Circles of Scientific Hell

First Circle: Limbo

Second Circle: Overselling

Third Circle: Post-Hoc Storytelling

Fourth Circle: P-Value Fishing

Fifth Circle: Creative Use of Outliers

Sixth Circle: Plagiarism

Seventh Circle: Non-Publication of Data

Eighth Circle: Partial Publication of Data

Ninth Circle: Inventing Data

P-Fishing

Fourth Circle: P-Value Fishing“Those who tried every statistical test in the book until they got a p value less than 0.05 find themselves here, an enormous lake of murky water. Sinners sit on boats and must fish for their food. Fortunately, they have a huge selection of different fishing rods and nets (brand-names include Bayes, Student, Spearman and many more). Unfortunately, only one in 20 fish are edible, so they are constantly hungry.”

P-Fishing

•Also known as…

▫Questionable Research Practices (QRPs))

▫Torturing the data▫P-Hacking▫Outcome reporting bias▫Undisclosed flexibility▫Researcher Degrees of Freedom▫…and more.

P-Hacking Works!

•Collect some data

P-Hacking Works!

•Collect some data•Try many statistical tests on the same

data

P-Hacking Works!

•Collect some data•Try many statistical tests on the same

data•Or try many variants of the same data

(e.g. removing ‘outliers’.)

P-Hacking Works!

•Collect some data•Try many statistical tests on the same

data•Or try many variants of the same data

(e.g. removing ‘outliers’.)•Or try looking at different variables within

the dataset

P-Hacking Works!

•Collect some data•Try many statistical tests on the same data•Or try many variants of the same data (e.g.

removing ‘outliers’.)•Or try looking at different variables within

the dataset•Report the analyses that give the most

favourable results (usually the lowest p-values).

Why Hacking Is So Useful

• There are many choices (‘researcher degrees of freedom’) in data analysis.

• For example, in a simple task-based fMRI data analysis, Joshua Carp found 7000 combinations of parameters (very conservative).

Carp, J. (2012).On the plurality of (methodological) worlds: estimating the analytic flexibility of fMRI experimentsFrontiers in Neuroscience

How Not To Get Caught

• There are many tools, but perhaps the most intuitive: the p curve

Simonsohn, U. Nelson, L. D. Simmons, J. P. (2013). P-curve: a key to the file-drawer. Journal of Exp. Psychol General

Try it now!http://www.p-curve.com/app2/

How It Can Be Stopped

Smulders YM (2013). A two-step manuscript submission process can reduce publication bias. Journal of Clinical Epidemiology

Chambers CD (2013). Registered Reports: a new publishing initiative at Cortex Cortex

Happy hacking!

@Neuro_Skeptic

neuroskeptic@googlemail.com

http://blogs.discovermagazine.com/neuroskeptic