Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group ›...
Transcript of Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group ›...
![Page 2: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/2.jpg)
Outline
1. ScienAficmethodandresearchfailures2. Definingreproducibleresearch3. Strategiesforreproducibility
![Page 3: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/3.jpg)
ObservaAon AskQuesAon
BackgroundResearch
FormHypothesis
DesignExperiment/
Study
CarryOutExperiment/
Study
Data
Analysis
Conclusions
ReportResults
Results
ThescienAficmethod
Sche
maA
ccourtesyofE
rikaMud
rak
1.ScienAficmethodandresearchfailure
![Page 4: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/4.jpg)
ConAnuumofresearchfailure
Disorganiza*on Egregiousbehavior
FailureofintegrityFailureofprocess
Deliberatemanipula.onofdatatogetresults-P-hacking-”Fishingexpedi.ons”
1.ScienAficmethodandresearchfailure
![Page 5: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/5.jpg)
ObservaAon AskQuesAon
BackgroundResearch
FormHypothesis
DesignExperiment/
Study
CarryOutExperiment/
Study
Data
Analysis
Conclusions
ReportResults
Results
P-hacking/fishingexpediAon
P<0.05
1.ScienAficmethodandresearchfailure
![Page 6: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/6.jpg)
ConAnuumofresearchfailure
Disorganiza*on Egregiousbehavior
FailureofintegrityFailureofprocess
Deliberatemanipula.onofdatatogetresults-P-hacking-”Fishingexpedi.ons”
HARK-ing
1.ScienAficmethodandresearchfailure
![Page 7: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/7.jpg)
P-hackyourwaytoscienAficglory
hVps://projects.fivethirtyeight.com/p-hacking/
1.ScienAficmethodandresearchfailure
![Page 8: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/8.jpg)
ObservaAon AskQuesAon
BackgroundResearch
FormHypothesis
DesignExperiment/
Study
CarryOutExperiment/
Study
Data
Analysis
Conclusions
ReportResults
Results
HypothesizingAYerResultsareKnown(HARK-ing)
1.ScienAficmethodandresearchfailure
![Page 9: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/9.jpg)
IsHARK-ingeverokay?
Exploratory Confirmatory
ResearchGoals
• Exploratoryresearch=hypothesisgeneraAon• Confirmatoryresearch=hypothesistesAng
1.ScienAficmethodandresearchfailure
![Page 10: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/10.jpg)
ConAnuumofresearchfailure
Disorganiza*on Egregiousbehavior
FailureofintegrityFailureofprocess
Deliberatemanipula.onofdatatogetresults-P-hacking-”Fishingexpedi.ons”
HARK-ing“Gardenofforkingpaths”
1.ScienAficmethodandresearchfailure
![Page 11: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/11.jpg)
Data
Analysis
ReportResults
Results
Thegardenofforkingpaths(GelmanandLoken,2013)
Thisobserva.onseemsfunny—shouldIthrowitout?
Imputemissingdata?
OtherstudiescontrolforX,somaybeIshouldaddthatin?
ShouldIlogtransformthis?
Logit,probitorlinearprobabilitymodel?
CanwereallyassumethatXisexogenous?Everyoneelsedoes.
Itriedthisthingbutitwasn’tsignificant,doIreportit?Thisdistribu.onlooksfunny—howcanIfixit?
Thoseresultsdidn’tmakesense,shouldIreportthemanyway?Towinsorizeornottowinsorize….
Myinterac.onisn’tsignificant…shouldItakeitout?
Thisobserva.onseemsfunny—shouldIthrowitout?
Imputemissingdata?OtherstudiescontrolforX,somaybeIshouldaddthatin?
ShouldIlogtransformthis?
Logit,probitorlinearprobabilitymodel?
CanwereallyassumethatXisexogenous?Everyoneelsedoes.
Itriedthisthingbutitwasn’tsignificant,doIreportit?
Thisdistribu.onlooksfunny—howcanIfixit?
Thoseresultsdidn’tmakesense,shouldIreportthemanyway?
Towinsorizeornottowinsorize….Myinterac.onisn’tsignificant…shouldItakeitout?
Imputemissingdata?
1.ScienAficmethodandresearchfailure
![Page 12: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/12.jpg)
ConAnuumofresearchfailure
Disorganiza*on Egregiousbehavior
FailureofintegrityFailureofprocess
Deliberatemanipula.onofdatatogetresults-P-hacking-”Fishingexpedi.ons”
HARK-ing“Gardenofforkingpaths”
CodingerrorsPoordocumenta.on
1.ScienAficmethodandresearchfailure
![Page 13: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/13.jpg)
Toavoidtheperilsofthegarden,HARK-ing,P-hacking,andsillymistakes…
• Integrity!-->Behonestwithyourself.• Transparency!-->Behonestwithyourreaders.• Doyoufeelgoodenoughaboutyourdecision-makingprocessestowritethemdownforalltosee?
Reproducibleresearch!
1.ScienAficmethodandresearchfailure
![Page 14: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/14.jpg)
Replicabilityvsreproducibility• Replicability
– EssenAaltothescienAficmethod– repeaAngastudyfromscratchusingnewdata,analystandcode– ifagivenrelaAonshipbetweenXandYistrue,itshouldshowupin
mulAplestudies
2.DefiningReproducibility
![Page 15: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/15.jpg)
ObservaAon AskQuesAon
BackgroundResearch
FormHypothesis
DesignExperiment/
Study
CarryOutExperiment/
Study
Data
Analysis
Conclusions
ReportResults
Results
Replicability2.DefiningReproducibility
![Page 16: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/16.jpg)
Replicabilityvsreproducibility• Reproducibility
– GefngtheexactsameresultasanexisAngstudyusingnewanalyst,butsamedataandcode
– RecentlytractableduetocompuAngandsoYwareadvances
2.DefiningReproducibility
![Page 17: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/17.jpg)
ObservaAon AskQuesAon
BackgroundResearch
FormHypothesis
DesignExperiment/
Study
CarryOutExperiment/
Study
Data
Analysis
Conclusions
ReportResults
Results
Reproducibility2.DefiningReproducibility
![Page 18: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/18.jpg)
Reproducibility
• FacilitatetransparencybycommunicaAngprocedureseasily
• IdenAfyinadvertenterrors• Avoidembarrassment• FacilitatecollaboraAon• SaveAme• GreaterpotenAalforextensionofwork-->higherimpactoverAme
2.DefiningReproducibility
![Page 19: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/19.jpg)
Thepublic/theintegrityofscience
Researchersinyourfield
Reviewers
Colleagues/Coauthors
Youin6months
Younextweek
You!
Whoareyouaccountableto?2.DefiningReproducibility
![Page 20: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/20.jpg)
Whatareweaimingfor?• SufficientdocumentaAontobringanunfamiliaruserupto
speed– Codebook– Readmefile– Variableandvaluelabelsinanalysisdataset– EffecAvecommentsincode
• Asingleclickexecutesyourprojectfromstarttofinish.– Downloading– Reformafng– CleaningandvariableconstrucAon– Analysis– Outputtables,graphs,figures– Reproduciblereport
2.DefiningReproducibility
![Page 21: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/21.jpg)
Howdowegetthere?
• Separatethephasesofdatawork• SystemaAcfileandnamingstructures• EffecAveandorganizedscripAng• Reproduciblereports
2.DefiningReproducibility
![Page 22: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/22.jpg)
Separatephasesofdatawork
1. Dataconversion/cleaning/variableconstrucAon
2. Analysis3. ReportgeneraAon
3.StrategiesforReproducibility
![Page 23: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/23.jpg)
NamingconvenAons• AgreewithyourcollaboratorsonnamingconvenAons.• Humanreadable
– Short,usefulnames– InformaAononcontent
• Machinereadable– Avoidspecialcharacters,spaces,etc
• CamelCase,ALLCAPS,lowercase,alloneword,underscore_between– Consistentnamingtofacilitatesearching
• Defaultordering– DateformatYYYYMMDD– Othernumbers—addleadingzeros
• Nevercallsomething“final”.Itprobablyisn’t.
3.StrategiesforReproducibility
![Page 24: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/24.jpg)
SystemaAcfilestructure
• Mustbecommontoallusers!• ChooseafilestructureandsAcktoit.• Makeskeletonoffolderswhenyoustartaproject.
3.StrategiesforReproducibility
![Page 25: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/25.jpg)
• /dta/original/stataraw/clean/analysis
• /documenta*on/metadata/reports
• /do/cleaning/analysismaster.do
• /output/figures/tables/oldoutput
• /wri*ng/paper1/paper2/notes/olddraYs
• /temp
Variable-ormodule-specificcleanfiles
Copyofread-onlyoriginalfilesexactlyasobtained.DataaYerconversiontoformatofchoice
Dataset(s)youwilluseforanalysis
Any/allcodebooksormetadatarelatedtodataCollecAonofdocumentswherethedatawasused,cited,described
Cleaning,merging,reshaping,variableconstrucAonscriptsAnalysisscripts
ScriptthatsetsuprelaAvefilepathsandcallsallscripts
SeparatefoldersifmulAplepapersusingthesamedata
OpAonalasneededKeepolderversionsofpaper,butgetthemoutoftheway
Keepforreference,ifyouchoose.
Subfoldersdependontypeofproject
GetridofcluVerasyoumakeit
![Page 26: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/26.jpg)
ScripAngAps• Data+Script=ReproducibleOutput• Masterscript:Runsotherscriptsincorrectorder• ModularscripAngvs.onebigfile
– Separatetypesofprocesses(cleaning,analysis)– AvoidrepeaAngblocksofcode:Separateprogramforrepeatedprocesses
• Notes/comments.– Consistentheaders– Usefulcomments,notexpressionsoffeeling
• Clarity>efficiency?Consideryourcollaborators.• Re-runscriptfromthebeginningregularly.Itmustrun!
3.StrategiesforReproducibility
![Page 27: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/27.jpg)
ReproducibleReports
• Integratecodeintotheproseofyourreport• Singlefilethatexecutesallstepsofdataprocessandoutputsafinalpaper
• Knowexactlywhatdatawasusedforanalysis,whatcodemadewhichfigure,etc.
• Disadvantages—learningcurve,iniAalinvestment.
• AlternaAvemethod:Copyandpaste.
3.StrategiesforReproducibility
![Page 28: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/28.jpg)
Avoidresearchfailuresbyimplemen*ngreproducibleresearchtechniquestoimprove
organiza*onandtransparency
1. Separatephasesofresearch2. SystemaAcfilenamingandstructure3. EffecAveandorganizedscripAng4. Reproduciblereports
• PrioriAzeelementsthatareaVainableforyou.
Yourfutureselfthanksyou!
![Page 29: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/29.jpg)
AddiAonalresources
• P-hackyourwaytoscienAficglory!hVps://projects.fivethirtyeight.com/p-hacking/
• GelmanandLoken(2013)GardenofForkingPaths.hVp://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
![Page 30: Reproducible Research - Cornell Universitybarrett.dyson.cornell.edu › research › group › Reproducibility... · 2019-06-08 · Reproducible research! 1. ScienAfic method and](https://reader033.fdocuments.us/reader033/viewer/2022053019/5f21ec0d76d8fc48b67400d5/html5/thumbnails/30.jpg)