Dominik Vogel How open is public administration …...2020/02/03 · Open science is “the process...

How open is public administration research and what should we change to be more open?

Dominik Vogel

@DrDominikVogel

Disclaimer

htt

ps:

//p

ixa

ba

y.co

m/p

ho

tos/

sch

oo

l-te

ach

er-

ed

uca

tio

n-a

sia

-178

2427

/S

top

sig

n: F

ree

pik

.co

m

03.02.2020 | Dominik Vogel 2

What is open science?

303.02.2020 | Dominik Vogel

Op

en

pe

erre

vie

wim

ag

e[m

od

ifie

d]:

Jo

e T

he

Go

at

Farm

er -

Ho

w t

o G

row

Yo

ur

Em

ail

Lis

t w

ith

A G

rea

t N

ewsl

ett

er, C

C-B

Y 2

.0

Log

o O

ER

: M

ark

us

Bü

sges

(le

om

ari

ad

esi

gn

) fü

r W

ikim

ed

ia D

euts

chla

nd

e. V

., C

C-B

Y-S

A 4

.0 v

ia W

ikim

ed

ia C

om

mo

ns.

Op

en

So

urc

e S

oft

wa

re: L

og

o O

pe

n S

ou

rce

Init

iati

ve [

mo

dif

ied

] b

y Si

mo

n P

hip

ps

un

de

r C

C-B

Y 2

.5

Open Peer Review …

https://creativecommons.org/licenses/by/2.0/

https://creativecommons.org/licenses/by-sa/4.0/

https://creativecommons.org/licenses/by/2.5/deed.en

Open science is “the process of making the content and

process of producing evidence and claims transparent

and accessible to others” (Munafò et al. 2017, p. 5).


OK. But why?


htt

ps:

//p

ixa

ba

y.co

m/p

ho

tos/

qu

est

ion

-ma

rk-w

hy-

pro

ble

m-s

olu

tio

n-2

1239

67/

Normative answer Practical answer


To

ols

ph

oto

by

Ha

up

es

Co

.on

Un

spla

sh

https://unsplash.com/@haupes?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText

https://unsplash.com/s/photos/tool?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText

The replication / credibility crisis in psychology


642

20

05

20

11

20

14

20

17

20

12

20

15

20

19

Bem: “Feelingthe Future”

Simonsohnet al: “p-Curve: The key to the file drawer”

Chambers: “The Seven Deadly Sins of Psychology”

Ioannidis: “Why most research findingsare false”

Simmons et al.: “False-

positive Psychology”

Open ScienceCollaboration:

“Estimating the Reproducibility

of Psych

Kvarven et al.: “Comparing

meta-analyses and prereg.

multi-lab repl. projects

based on Spellman et al. 2017

De

sig

ne

db

yP

rese

nta

tio

nG

O.c

om

Why do we have such low

replicability?

What results can we trust?

Psychologists (and other social scientists) wonder


Ima

ge

by

Ro

bin

Hig

gin

sfr

om

Pix

ab

ay

https://pixabay.com/users/RobinHiggins-1321953/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=3355958

https://pixabay.com/

At the center: Publication

bias and the file drawer

problem

The many ingredients of the replication crisis


htt

ps:

//p

ixa

ba

y.co

m/p

ho

tos/

file

s-p

ap

er-

off

ice

-pa

pe

rwo

rk-s

tack

-16

1422

3/

Preference for novel, surprising, and significant results

sets incentives for Questionable Research Practices

(QRP)

HARKing: Hypothesizing after results are known

p-hacking: additional analyses / data to pass p < .05

Conducting underpowered studies

Fraud

The many ingredients of the replication crisis


Ph

oto

by

Ca

lum

Lew

iso

n U

nsp

lash

https://unsplash.com/@calumlewis?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText

https://unsplash.com/s/photos/ingredients?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText

Transparency:

Everybody should be able to assess how results were obtained

Reducing researcher degrees of freedom

Define as much as possible in advance

The solution(?)


Open Data

Open Materials (esp. Code)

Reporting standards

Open peer review

Open Source Software

[Open Access]

The solution: Transparency


Op

en

pe

erre

vie

wim

ag

e[m

od

ifie

d]:

Jo

e T

he

Go

at

Farm

er -

Ho

w t

o G

row

Yo

ur

Em

ail

Lis

t w

ith

A G

rea

t N

ewsl

ett

er, C

C-B

Y 2

.0

Op

en

So

urc

e S

oft

wa

re: L

og

o O

pe

n S

ou

rce

Init

iati

ve [

mo

dif

ied

] b

y Si

mo

n P

hip

ps

un

de

r C

C-B

Y 2

.5

https://creativecommons.org/licenses/by/2.0/

https://creativecommons.org/licenses/by/2.5/deed.en

Separate exploratory from confirmatory research

Confirmatory: Define as much as possible in advance

Less ways to (unconsciously) tweak the results in the

desired direction

Preregistration

Registered reports

The solution: Reducing researcher degrees of freedom


Culture needs to change

Incentives need to change

Tools do not magically lead to better science


Ima

ge

by

Ale

xa

s_Fo

tos

fro

m P

ixa

ba

y

https://pixabay.com/users/Alexas_Fotos-686414/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=2574251

https://pixabay.com/?utm_source=link-attribution&utm_medium=referral&utm_campaign=image&utm_content=2574251

Open science badges

Registered reports

Journals value replications

Many Lab projects / large-scale replications

Error (and fraud?) detection

Changes to incentive structure


Enough psychology, I want to learn about PA


Sto

psi

gn

: Fre

ep

ik.c

om

Yes

No careful assessment, yet

Incentives are the same as

in psychology

Survey research offers even

more ways for HARKing

and p-hacking (control

variables)

No

No evidence

Less small-n experiments

(yet?)

Less ways to repeat studies

to get intended results

More PSM of PA

researchers(?)03.02.2020 | Dominik Vogel 17

Is there a replication crisis in PA?

Ico

ns

ma

de

by

Fre

ep

ikfr

om

ww

w.f

lati

con

.co

m

https://www.flaticon.com/authors/freepik

http://www.flaticon.com/


Does the literature on the PSM–

performance relationship

contain evidential value?

(Vogel & Homberg under review)

Distribution of p values (p-curve) follows a predictable

pattern

Holds for subset of significant p values

Reporting of significant p values should be unbiased

p-curve method: analyze significant p values of published research


Distribution of p values without a true effect



not p-hacked p-hacked

Result of the p-curve analysis


So, no reason to worry?


Ph

oto

by

Lid

yaN

ad

ao

n U

nsp

lash

https://unsplash.com/@lidyanada?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText

https://unsplash.com/s/photos/happy?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText

We know little about the credibility of PA research

Even if there is no replication crisis, open science

practices help to prevent a crisis in the future

They help to do better science find the truth

Why should PA adopt open science practices?


Reviewers are more aware of adverse effects of

underpowered studies, HARKing, and p-hacking

Preregistration more and more common and valued

New open access journals

Funders are pushing for open

science practices

What did already change?


No pre-print culture

No registered reports

Journals still closed access

What did not change?


“You’re doing it because you want to do

high quality work. You want to have the

best possible chance of learning

something True about the world and the

people in it.” (Corker 2018)

“The first principle is that you must not

fool yourself – and you are the easiest

person to fool.” – Richard Feynman

Why should I adopt open science practices?


Ph

oto

of

Ric

ha

rd F

eyn

ma

n b

yT

am

iko

Th

iel a

vail

ab

leu

nd

er

a C

C B

Y-S

A 3

.0li

cen

se

https://creativecommons.org/licenses/by-sa/3.0/deed.en

Preregister your studies when possible (and indicate

exploratory work)

Publish your data, analysis code, and materials

When reviewing: ask for proper reporting and

transparency; be skeptical

Publish the accepted manuscripts of your publications

Publish pre-prints?

OK, you convinced me. What can I do?


Require proper statistical reporting

Enable registered reports

Push for open data, open materials

Encourage pre-prints

Encourage replications

Adopt TOP guidlines

What can journals do?


Value open science practices

Switch from traditional publishing

system to open access

What can societies do?


Prof. Dr. Dominik Vogel

University of Hamburg

Assistant Professor of Public Management

Von-Melle-Park 9

20146 Hamburg, Germany

[email protected]

Twitter: www.twitter.com/DrDominikVogel

Website: https://vogel-online.info/en


Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425. https://doi.org/10.1037/a0021524

Chambers, C. (2017). The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton, Oxford: Princeton University Press.

Corker K. (2018). Open Science is a Behavior. https://cos.io/blog/open-science-is-a-behavior/

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

Kvarven, A., Strømland, E., & Johannesson, M. (2019). Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nature Human Behaviour. Advance online publication. https://doi.org/10.1038/s41562-019-0787-z

Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S.,

Chambers, C. D., Du Percie Sert, N., . . . Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 21. https://doi.org/10.1038/s41562-016-0021

Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology. General, 143(2), 534–547. https://doi.org/10.1037/a0033242

Spellman, B., Gilbert, E. A., & Corker, K. S. (2017). Open Science: What, Why, and How. https://doi.org/10.31234/osf.io/ak6jr


Literature

https://doi.org/10.1037/a0021524

https://cos.io/blog/open-science-is-a-behavior/

https://doi.org/10.1371/journal.pmed.0020124

https://doi.org/10.1038/s41562-019-0787-z

https://doi.org/10.1126/science.aac4716

https://doi.org/10.1177/0956797611417632

https://doi.org/10.1037/a0033242

https://doi.org/10.31234/osf.io/ak6jr

Dominik Vogel How open is public administration …...2020/02/03 · Open science is “the process...

Documents

Transcript of Dominik Vogel How open is public administration …...2020/02/03 · Open science is “the process...