Code publication and peer review Peer review...Reviewers are provided private access to the capsule...
Transcript of Code publication and peer review Peer review...Reviewers are provided private access to the capsule...
Code publication and peer reviewa discussion of best practices, technological innovations and recognition
Erika Pastrana, Editorial Director,
Nature Journals
An
tarctica meltd
ow
n co
uld
do
ub
le sea level rise
Outline
What problem are we trying to solve?
Best practices and guiding principles
Technological innovations---Trialing container platforms for peer review and publication of code. Results from Code Ocean-Nature journal trial.
Incentives and Recognition
Discussion
2
Nature Research PowerPoint presentation title / 00.00.2017
2
1.0
What problem are we trying to solve and what are our guiding principles?
Today’s scientific discovery process depends on a dynamic set of digital resources: data, code and the right environment to run the analysis.
The number of research papers that make use or develop new code is increasing rapidly.
When code is central to the main claims in the paper, it is imperative that the code be properly documented, evaluated so that it is functional, and permanently accessible to ensure reproducibility of the results.
As publishers, we want to adapt to the changing outputs of science, improve our papers to increase reproducibility and open science, and lead in innovation.
What problem are we trying to solve?
4
Nature Research PowerPoint presentation title / 00.00.2017
Best practice and guiding principles
• Proper documentation : Code needs to be sufficiently documented to enable others to check and re-use it. This includes information on dependencies, operating systems, technical requirements as well as licenses and terms of use.
• Peer review: peer reviewing the code ensures that it is evaluated by an expert and it is functional and re-usable.
• Sharing and recognition : as the code is a central element of the paper and the research output, it should be shared and cited via a permanent identifier (DOI) in a form that facilitates wider use by the academic community and recognition of its own value.
5
Nature Research PowerPoint presentation title / 00.00.2017
5
2.0
Technological innovations
A case study: Nature Journals trial the use of container platforms for peer review and publication of code
• Currently, the Nature journals mandate that the availability and conditions of access of custom code that is central to the paper, be explicitely stated.
• For over 10 years, several Nature journals (eg Nature Biotechnology, Nature Methods)have been asking authors to submit the code for peer review and asking reviewers to check it by installing and running the code. These journals also mandate that the code be shared in the published the paper.
Several Nature journals have a long history in making code accessible and robust upon publication
Reproduced from Pastrana, E., Kousta, S. & Swaminathan, S. Three Approaches to Support Reproducible Research. Science Editor
7
Nature Research PowerPoint presentation title / 00.00.2017
But….…publishing reproducible code is cumbersome and time consuming
Reviewers have to…
• Find the code• Acquire the right hardware• Set up the environment• Import the right data files• Install dependencies, packages, versions, operating systems…• Errors… fix… errors…• Run (on own compute time)• Verify code runs and produces results
+
• Contrast with claims in the paper
8
Nature Research PowerPoint presentation title / 00.00.2017
http://blogs.nature.com/ofschemesandmemes/2018/08/01/nature-research-journals-trial-new-tools-to-enhance-code-peer-review-and-publication
A trial to test the use of container platforms for peer review and publication of code
We want to adopt improved technologies for peer review and publishing of code:
Software container tools, such as Code Ocean (https://codeocean.com/) are cloud-based platforms that host the code, data and environments necessary to re-run the code and reproduce the results shown in the paper.
Users can run the code with the set parameters to enable reproducing the results or execute the code with new input values.
We ran a trial (August 2018 to Sept 2019) on Nature Methods, Nature Biotechnology and Nature Machine Intelligence in partnership with Code Ocean, to test the usefulness of code containers (aka ‘code capsules’) for peer review and publication of code.
Trial details:
Authors are given the option to set up a Code Ocean capsule when paper goes out to review to facilitate compliance with code peer review and publication policies.
Reviewers are provided private access to the capsule and free-computing time.
Readers access code, data and environment in one place, via a link to the capsule. Capsule is given a DOI to enable proper recognition, citation and re-use of code.
A trial to test the use of container platforms for peer review and publication of code
11
Nature Research PowerPoint presentation title / 00.00.2017
Reproduced from Pastrana, E., Kousta, S. & Swaminathan, S. Three Approaches to Support Reproducible Research. Science Editor
12
Nature Research PowerPoint presentation title / 00.00.2017
12
3.0
What have we learnt from the trial?
13
Nature Research PowerPoint presentation title / 00.00.2017
What we set out to monitor:KPI WhoMonitoring author opt-in to the trial and asking for feedback from authors if they decline. Editors
Effects on peer review: TAT from submission to first decision and TAT for CO capsule set up services
Editors/Code Ocean
Effects on peer review: qualitative assessments on the ‘quality’ of peer review of code Editors
Survey authors and referees involved in the trial to monitor author and referee satisfaction with the user interface, level of Code Ocean support and overall feedback on the use of the platform for peer review.
Editors
Engagement with Code Ocean capsule: Code OceanNumber of duplications of the compute capsule by reviewers Code Ocean
Total runs of compute capsules (pre publication) Code Ocean
Total runs of compute capsules (post publication) Code Ocean
Number of compute capsule views (post publication) Code Ocean
Total duplications/downloads of compute capsules (post publication) Code Ocean
Total numbers of citations to the code’s DOI (post publication) Code Ocean
Results: We now ensure our papers provide open, verified, properly
documented and cited code
Results:We see high author uptake despite the additional work required
Papers that participated in the trial >120 papers in the trial from the three journals
Papers published:>20 papers published that have used the functionality for peer review and publication of code
Author uptake:
54% of all authors from the three journals opted in to the trial, with 59% opting in at Nature Biotechnology, 44% at Nature Methods and 60% at Nature Machine Intelligence
16
Nature Research PowerPoint presentation title / 00.00.2017
Reviewer engagement :
On average private links were viewed 34 times. 45% of the capsules received a reviewer sign up/duplication On average, each reviewer that signed in run the code 1.3 times.
Reader engagement:
Published Code Ocean capsules associated with Nature papers were viewed on average 122 times and were run 14 times
(*) data from code ocean site.
Results: We see high reviewer and reader engagement with the platform
Reasons for opt out:
• 20% of authors who opted out did so because they felt their current platform (typically GitHub) already complied with our policies for code peer review
• 23% of authors thought Code Ocean would not be a suitable platform for their paper*
• 10% of authors feared that setting up a capsule would delay publication*
• 40% of authors who opted out gave no reasons
(*) potential to be reduced with better communication
Results: The main reasons for opt out are that authors are using GitHub
or think the platform won’t be suitable for them
Reviewer & Author comments
“I found the Code Ocean platform an excellent tool for reviewing the code thanks to its easy environment setup and clear instructions for running and data analysis. In order to review the code, I also carefully read the manuscript, so I offer my remarks for both”
“In addition to reviewing the manuscript, I tested its code and sample data through the Code Ocean container platform. I found the files and documentation provided with the package to be thorough, and was able to run the program without issue. Further, it was well-documented and easy to use (especially in the Code Ocean container system) and that will hold serious appeal to readers who might consider implementing this tool.”
“I do not have any experience with Code Ocean as a reviewer, but I would like to state that my experience with it as an author, it was straightforward for use by Authors and they were responsive and helpful when we reached out to them. So I have positive experience with them in that respect.”
“This is an interesting new tool, with useful features and reasonable amount of free computing time. It provides a great platform for code examination and dissemination, and should greatly contribute to reproducibility and open science.”
Results: We receive strong support for the initiative
19
Nature Research PowerPoint presentation title / 00.00.2017
Next steps and more information
We will continue with this initiative as a standard editorial practice going forward at
the existing Nature journals and are looking to expand this practice to more journals.
Nature Biotechnology: ‘Changing coding culture’
Nature Machine Intelligence: ‘Sharing high expectations’
Nature Methods: ‘Easing the burden of code review’
Science Editor: Three approaches to support reproducible research
Of Schemes and Memes: What we have learnt testing container platforms for peer
review and publication of code
Code Ocean blog post: Peer review on Code Ocean
20
Nature Research PowerPoint presentation title / 00.00.2017
20
4.0
Incentives and Recognition
21
Nature Research PowerPoint presentation title / 00.00.2017
Recognition
https://www.niso.org/press-releases/2019/01/reproducibility-taxonomy-definitions
• We would like to encourage best practice among the community by awarding papers that provide verified/peer reviewed code a ‘badge’ that follows community standards
• Badges will promote sharing code and will signal when code has been verified/peer reviewed
• We are part of a NISO working group developing Badge-recommendations for the computational and Computing sciences
22
Nature Research PowerPoint presentation title / 00.00.2017
22
The story behind the image
Antarctica meltdown could double sea level rise
Researchers at Pennsylvania State University have been considering how quickly a glacial ice melt in Antarctica would raise sea levels. By updating models with new discoveries and comparing them with past sea-level rise events they predict that a melting Antarctica could raise oceans by more than 3 feet by the end of the century if greenhouse gas emissions continued unabated, roughly doubling previous total sea-level rise estimates. Rising seas could put many of the world’s coastlines underwater or at risk of flooding and storm surges.
Thank you
Erika Pastrana, PhD
Editorial Director, Nature Journals
@ErikaPastrana1