The Registered Reports project: A vaccine against research ... · Are the conclusions jus’ﬁed...

TheRegisteredReportsproject:Avaccineagainstresearchbias?

Chris Chambers School of Psychology, Cardiff University

Email: [email protected] Twitter: @chrisdc77 1

What’sbestforscience

Highqualityresearch,regardlessofoutcome

What’sbestforscien:sts

Producingalotofpublishableresults

Sciencehasanincen:veproblem

seeNosek,Spies&Motyl(2012).Perspec'vesonPsychologicalScience,7(6):615–631

Generate and specify hypotheses!

Design study!

Collect data!Analyse data & test hypotheses!

Interpret data!

Publish or conduct next experiment!

Hypothe:co-deduc:vescien:ficmethod

Publication bias!Lack of data sharing!

Low statistical power!

Significance chasing!

Significance chasing!

Lack of replication!

1in1000papersMakeletal(2012)

~50%chancetodetectmediumeffectsCohen(1962);SedlmeierandGigerenzer(1989);BezeauandGraves(2001)

~50-100%prevalenceJohnetal(2012)

~50-90%prevalenceJohnetal(2012)Kerr(1998)

~92%posiWveFanelli(2010)

~70%failureWichertsetal(2006)

Whyisthishappening?

Becauseweplacetoomuchimportanceontheresultsofexperimentsandnotenoughontheprocessesthatproducethem

ResultsmakescienceexciWngbutjudgingthequalityofscience(andscienWsts)accordingtotheresultsis“so`”science

Canwefixthis?YesPhilosophy:Whatgiveshypothesis-tesWngitsscienWficvalueis•  theQUESTIONitasks•  theQUALITYofthemethodituses•  nevertheRESULTitproduces

Ifweacceptthisphilosophytheneditorialdecisionsatjournalsshouldbeblindtoresults

Thisisnotanewidea

“Whatweneedisasystemforevalua'ngresearchbasedonlyontheproceduresemployed.Iftheproceduresarejudgedappropriate,sensible,andsufficientlyrigoroustopermitconclusionsfromtheresults,theresearchcannotthenbejudgedinconclusiveonthebasisoftheresultsandrejectedbytherefereesoreditors.Whethertheprocedureswereadequatewouldbejudgedindependentlyoftheoutcome.”

RobertRosenthal(1966).Experimentereffectsinbehavioralresearch.NewYork.

RegisteredReports

FourcentralaspectsoftheRegisteredReportsmodel:

•  Partofthepeerreviewprocesstakesplacebeforeexperimentsareconducted

•  PassingthisstageofreviewvirtuallyguaranteespublicaWon•  Originalstudiesandhigh-valuereplicaWonsarewelcome

•  Researchersdecidehypotheses,experimentalprocedures,andmainanalysesbeforedatacollecWon

AuthorssubmitSTAGE1manuscriptwithIntroducWon,ProposedMethods&

Analyses,andPilotData(ifapplicable)

Stage1peerreview

IfreviewsareposiWvethenjournaloffersin-principleacceptance(IPA),

regardlessofstudyoutcome(protocolnotpublishedyet)

Howitworks

Arethehypotheseswellfounded?Arethemethodsandproposedanalysesfeasibleandsufficientlydetailed?Isthestudywellpowered?(≥90%)Havetheauthorsincludedsufficientposi'vecontrolstoconfirmthatthestudywillprovideafairtest?

Howitworks

Stage2peerreview Didtheauthorsfollowtheapprovedprotocol?Didposi'vecontrolssucceed?Aretheconclusionsjus'fiedbythedata?Manuscriptpublished!

Authorsdotheresearch

AuthorsresubmitcompletedSTAGE2manuscript:•  Introduc:onandMethods(virtuallyunchanged)•  Results(new):Registeredconfirmatoryanalyses

+unregisteredexploratoryanalyses•  Discussion(new)•  Datadepositedinapublicarchive

Noneofthesethingsmaper

hpp://www.journals.elsevier.com/cortex/virtual-special-issues/virtual-special-issue-registered-reports

PublishedexamplesatCortex

Seealso:SocialPsychologyspecialissue:hpp://econtent.hogrefe.com/toc/zsp/45/3PerspecWvesonPsychologicalScience:hpp://www.psychologicalscience.org/index.php/replicaWon/ongoing-projects

–Reproducible–•  detailed,repeatablemethods•  highstaWsWcalpower(2-3xabovenormal)

–Transparent–•  accompaniedbyopendata&materials•  outcomesofconfirmatoryandexploratory

analysesdisWnguished

–Credible–•  nopublicaWonbias•  nohindsightbias•  noselecWvereporWng

Permanentadopters

Specialissues

Forfulllistseehpps://cos.io/rr/

RegisteredReportsatRoyalSocietyOpenScience

NowavailableinallSTEMareas,fromphysicstopsychology

hpp://rsos.royalsocietypublishing.org/content/registered-reports

14

RegisteredReportsatNatureHumanBehaviour

Whatarethebenefitsforjournals,editors,authorsandthescienWficcommunity?

Benefits

Benefits

1.  NopublicaWonbias

Benefits

1.  NopublicaWonbias2.  Logicallyeliminatesvariousforms

researcherbias(p-hacking,posthochypothesising)

Benefits



3.  HighstaWsWcalpowerrequirements

increasereproducibility

Benefits




increasereproducibility4.  IncenWvizesimportantreplicaWon

studiesandothernovel,resource-intensiveprojects(wherepublicaWonwouldnormallybeconWngentonresults)

Benefits




increasereproducibility4.  IncenWvizesimportantreplicaWon

studiesandothernovel,resource-intensiveprojects(wherepublicaWonwouldnormallybeconWngentonresults)

5.  Incorporatespublicarchivingofdata

andmaterials

FrequentlyaskedquesWons

28

1.“AreRegisteredReportssuitableformyfield?”

•  Applicabletoanyfieldengagedinhypothesis-drivenresearchwhereoneormoreofthefollowingproblemsapply:

•  Publica:onbias•  Significancechasing(e.g.p-hacking)•  Posthochypothesizing(hindsightbias)•  Lowsta:s:calpower•  Lackofdirectreplica:on

•  Notapplicablefor•  Purelyexploratoryscience•  Methodsdevelopment } NohypothesistesWng

3.“What’stostopRegisteredReportsfrombecomingadumpinggroundforinconclusivenullresults?”

•  aprioripowerrequirements(≥90%)increasereproducibilityofallfindings•  BayesianinferenWalmethodswelcomedforprovidingevidenceinfavourofH0orH1.

2.“Couldresearcherscheatby‘pre-registering’astudythattheyhavealreadyconducted?”•  Time-stampedrawdatafilesmustbesubmipedatStage2withbasiclablogand

cerWficaWonfromallauthorsthatdatawascollecteda`erprovisionalacceptance•  SubmiwngacompletedstudyatStage1wouldthereforebefraud•  StrategywouldbackfireanywaywhenreviewersaskforamendmentsatStage1

4.“Pre-registraWonisfineforseniorresearcherswhohave‘madeit’butI’majuniorscienWstandneedtoplaythegame”

•  Goingforpostdocjobs,whatyoudothinkwilllookbeperonyourCV?A)Bunchofpaperslistedas“inpreparaWon”,“submiped”B)Bunchofpaperslistedas“provisionallyacceptedat[respectedjournal]”

•  Thegameischanging:journalpoliciesarechangingtovaluetransparencyandreproducibility,e.g.NatureHumanBehaviourhaslaunchedRRs

RegisteredReportsaren’tdesignedtopreventfraudbuttoincen'vizegoodprac'ce

5.“WillthislimitexploraWonorsWgmaWzeexploratoryresearch?”•  No.Thearenorestric:onsontherepor:ngofunregisteredexploratoryanalyses.•  Confirmatoryandexploratoryanalysesaresimplyreportedseparatelyinthefinalpaper

•  ExploratoryReportsatCortex(indevelopment)•  nohypothesistesWng•  nopvalues•  Data-led;lightonintroducWonandtheory•  Purposeistogeneratehypothesesratherthantestthem

WhatsWgmaWzesexploratoryresearchisposthochypothesizingtofitadeducWveframeworkExploratoryresearchissimplynotvaluedinitsnaWveform

31

6.“Whathappensifweneedtochangesomethingaboutourexperimentalproceduresa`ertheyareprovisionallyaccepted?”

•  Minorchanges(e.g.replacingequipment)canbefootnotedinStage2manuscriptasprotocoldeviaWons

•  Majorchanges(e.g.changingdataexclusioncriteria)arelikelytorequirewithdrawal•  EditorialteamdecideswhetherdeviaWonissufficientlyminortoconWnue

8.“IhaveaccesstoanexisWngdatasetthatIhaven’tyetanalysed.CanIsubmitthisproposedanalysisasaRegisteredReport?”

•  NotatCortex,butotherjournalsofferthis,suchasEuropeanJournalofNeuroscience…

7.“Someofmyanalyseswilldependontheresults,sohowcanIpre-registereachstepindetail?”(e.g.typeofstaWsWcalmodel)

•  Pre-registraWondoesn’trequireeachdecisiontobespecified,onlythedecisiontree•  Authorscanpre-registertheconWngencies/rulesforfuturedecisions

hpps://docs.google.com/spreadsheets/d/1D4_k-8C_UENTRtbPzX}jEyu3BfLxdOsn9j-otrO870/edit#gid=0

9.“HowwillRegisteredReportsincenWvizereplicaWonstudies?”

•  Conspiracyofcircumstancestellsusnottobotherdoingdirect(exact)replicaWons

•  MethodsecWonsareo`entoovaguetoallowprecisereplicaWon•  ChroniclackofpowerinnovelresearchmeansthatreplicaWonso`en

requireverylargesamplessizes•  ApempWngtoexactlyrepeatapreviousexperimentcanbeseen(in

psychology)asanactofaggression(cf.physics)•  MoWvatedreasoningbyreviewerscanimpedepublicaWon•  Mostpsych/neurojournalswantnoveltyandseereplicaWonsas(usually)

unpublishable

•  RRs:haveproposedreplicaWonexperimentreviewedandprovisionallyacceptedbeforeyouinvestsubstanWalresourcesintodoingit;potenWallyinvolveoriginalauthorsinpeerreviewoftheprotocol;mo:vatedreasoningisprevented

11.“RegisteredReportsseemslimitedtosinglestudies.Butourpapersusuallyincludesequencesofexperiments”

•  WewelcomesequenWalregistraWonsinwhichauthorsaddstudiesiteraWvelyatStage1viaafast-trackmechanismandcompletethematStage2

•  Witheachcompletedcycle,thepreviousacceptedversionofthepaperisguaranteedtobepublished

•  AuthorscanalsoincludeasequenceofunregisteredexperimentsaspreliminarystudiesinaStage1RR(e.g.E1,E2,E3preliminary;manuscriptproposesE4aspre-registeredtest)

10.“Reviewerscouldstealmyideasatthepre-registraWonstageandscoopme”

•  OnlyahandfulofpeopleknowabouteachStage1submission•  OnceStage1protocolisaccepted,thejournalcan’trejectyourpaperbecause

somethingsimilarwaspublished(noveltyisirrelevant)•  ManuscriptreceiveddateonpublishedRRwillbethedateofStage1submission•  HowdifferentfromgrantapplicaWons,conferencepresentaWons,seminars?

35

Twothingswe’velearnedaseditors

1.ReviewerssomeWmesshi`thegoalpostsoncedataareinCasestudy:•  A`ermulWpleroundsofreview,areviewerapprovedprotocolatStage1•  Whenresultsfailedtoconfirmreviewer’sexpectaWonsatStage2,

reviewerraisednewmethodologicalobjecWons&apemptedtoreject

Casestudy:•  A`ermulWpleroundsofreview,areviewerapprovedprotocolatStage1•  WhenresultswerestaWsWcallynon-significant,reviewerdemandedthat

authorsconductalonglistofposthocanalysesto“findsomething”

EDITORALDECISION:Reviewerwasoverruled.BarringextremecaseswhereallparWes(authors,reviewers,editors)agreethatacriWcalflawwasoverlooked,objecWonstoStage1methodsareineligibleatStage2.LimitaWonsinsteadcoveredinDiscussion.

EDITORALDECISION:Posthocanalysescanonlyberequiredifdeemednecessarytosupportauthor’sconclusions.Authorinvitedtoconsiderextraanalysesbutnotrequiredtodothem.Reviewerinvitedtoconductanalysesusingopendataandpublishaseparatecommentpiece.

Upshot:RRsarerevealingreviewerbiasinwaythatisinvisibleinconvenWonalreview

36

Twothingswe’velearnedaseditors

2.LackofposiWvecontrolsinpsychologyandcogniWveneuroscience

•  ButfewiniWalsubmissionsproposesuchtests•  Manyfieldshavenosuchtests

STAGE1CRITERION6Whethertheauthorshaveconsideredsufficientoutcome-neutralcondiWonsforensuringthattheresultsobtainedareabletotestthestatedhypotheses

STAGE2CRITERION1Whetherthedataareabletotesttheauthors’proposedhypothesesbypassingtheapprovedoutcome-neutralcriteria

PrevailingassumpWon:astudyissaidtohave“worked”ifthemainhypothesiswassupported(p

37

Goingevenfurther…

CanweintegrateclinicaltrialregistraWon(whereitapplies),ethicalreview,grantfundingandRegisteredReports?

•  PossiblesoluWon:RegisteredReportsfundingmodel•  Authorssubmittheirresearchproposalbeforetheyhavefunding.

•  Followingsimultaneousreviewbytheboththefunderandthejournal,thestrongestproposalswouldbeofferedfinancialsupportbythefunderANDin-principleacceptanceforpublicaWonbythejournal.

Informa:onHubattheCenterforOpenScience

hpps://cos.io/rr/

Formoreinfo,emailme([email protected])orDavidMellorattheCOS([email protected])

•  DetailedFAQs•  Tablecomparingjournalfeatures

The Registered Reports project: A vaccine against research ... · Are the conclusions jus’ﬁed...

Documents

Transcript of The Registered Reports project: A vaccine against research ... · Are the conclusions jus’ﬁed...