Laurie Goodman at #CSE2014: Reproducibility: It's going to cost you time and effort, but it's our...
-
Upload
gigascience-bgi-hong-kong -
Category
Science
-
view
107 -
download
2
description
Transcript of Laurie Goodman at #CSE2014: Reproducibility: It's going to cost you time and effort, but it's our...
Reproducibility
Laurie Goodman, PhDEditor-in-Chief, GigaScience
Mostly Time & Effort
It’s Going to Cost You
But It’s Our Job
Growing Issue: increasing number of retractions>15X increase in last decade
Strong correlation of “retraction index” with higher impact factor
1. Science publishing: The trouble with retractions http://www.nature.com/news/2011/111005/full/478026a.html2. Retracted Science and the Retraction Index ▿ http://iai.asm.org/content/79/10/3855.abstract?
At current % increase by 2045 as many papers published as retracted!
Problem: Growing Replication Gap
1. Ioannidis et al., (2009). Repeatability of published microarray gene expression analyses. Nature Genetics 41: 142. Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8)
Out of 18 microarray papers, resultsfrom 10 could not be reproduced
The problems with publishing
• Scholarly articles are merely advertisement of scholarship . The actual scholarly artefacts, i.e. the data and computational methods, which support the scholarship, remain largely inaccessible --- Jon B. Buckheit and David L. Donoho, WaveLab and reproducible research, 1995
• Core scientific statements or assertions are intertwined and hidden in the conventional scholarly narratives
• Lack of transparency, lack of credit for anything other than “regular” dead tree publication
GigaSolution: deconstructing the paperNeed to credit and reward:
• Data/software availability
• Metadata/curation
• Interoperability
• Availability of workflows
• Transparent analyses
Data
Metadata
Methods
Analyses
Every ComponentHas a Citable DOI
How are we supporting reproducibility?
Data Sets inGigaDB
Analyses inGigaGalaxy
Paper inGigaScience
Linked to
Linked to
Open-access journal Data Publishing Platform
Data Analysis Platform
Example in Neuroscience
• Neuroscience Data are not typically shared
• Data AND Tools are not typically made available to the reviewers
• Journal Editors think Reviewers will not want to review data
GigaScience 2014, 3:3 doi:10.1186/2047-217X-3-3
Example in Neuroscience• Neuroscience Data are not typically shared• Author Dr. Stephen Eglen said: “One way of encouraging neuroscientists to
share their data is to provide some form of academic credit.”• We hosted with a DOI: 366 recordings from 12 electrophysiology datasets• GigaDB is included in Thompson Reuters Data Citation Index • Data AND Tools are not typically made available to the reviewers• We made manuscript, data and tools all available to the reviewers.• We make sure to include reviewers who are able to properly assess the data
itself and rerun the tools • To reduce burdens- we sometimes select a reviewer who ONLY looks at the
data.• Journal Editors think Reviewers will not want to review data• What Reviewer Dr. Thomas Wachtler said: “The paper by Eglen and
colleagues is a shining example of openness in that it enables replicating the results almost as easily as by pressing a button.”
• What Reviewer Dr. Christophe Pouzat said: “In addition to making the presented research trustworthy, the reproducible research paradigm definitely makes the reviewers job more fun!”
Reviewer Comments on the Process
http://www.biomedcentral.com/biome/christophe-pouzat-and-thomas-wachtler-on-reproducible-research-in-neuroscience/http://blogs.biomedcentral.com/gigablog/2014/04/16/qa-on-dynamic-documents/
Reviewer Take Home Message
http://www.biomedcentral.com/biome/christophe-pouzat-and-thomas-wachtler-on-reproducible-research-in-neuroscience/http://blogs.biomedcentral.com/gigablog/2014/04/16/qa-on-dynamic-documents/
But This Can’t Be Done For Wet Bench!!!
And Make All Data AvailableTake steps to put all the images and raw data in a repository,
and to use biobanking where possible
Address Insufficient-Method Syndrome:-Complete, Easily found, Searchable, and Updatable
Lessons learned:• It is possible to recreate a result from a paper
• Reproducibility is COSTLY. How much are you willing to spend?
• Learn a huge amount about the study, and provides lots of information not present in the paper
• Much easier to do this before rather than after publication
It’s Time to Move Beyond Dead Trees
18121665 1869
Thanks to:
[email protected]@gigasciencejournal.com
@gigascience
facebook.com/GigaScience
blogs.openaccesscentral.com/blogs/gigablog/
Contact us:
Scott Edmunds, Executive EditorNicole Nogoy, Commissioning EditorPeter Li, Lead Data ManagerChris Hunter, Lead BioCuratorRob Davidson, Data ScientistXiao (Jesse) Si Zhe, Database DeveloperAmye Kenall, Journal Development Manager
Follow us:
www.gigasciencejournal.comwww.gigadb.org