The state of scientific publishing and a new model

Dwight KravitzChris Baker

Motivation: Improving Cost and Quality

Budget per Paper ($1000)N

IH B

ud

get

/ #

Pap

ers

Pu

blis

hed

Peer Review: Outline

• History and evolution

• Modern purpose

• Improving peer review

• Post-publication

History of Peer Review and Publishing

• Problems facing authors– Disseminating results– Claiming ownership

of results

• Publishing– Publishers had

distribution channels– Provides a record of

when results were published

History of Peer Review and Publishing

• Problems facing publishers– Establishing veracity

of results– Deciding which results

to publish given limited space

Henry Oldenburg, 1665Medical Essays and Observations, 1731

• Peer review– Stamp of scientific

approval– Prioritizes papers for

inclusion by publishers

55% Reject42% Revise3% Accept

55% Reject42% Revise3% Accept

Given Revision:98% Accept2% Reject

64% of papers are rejected at least

once

Direct Costs of the Current System

• Total Number of Reviews: 6.3 (2-15)• Total Days under Review: 122 (21-

321)• Total Hours for Revisions: 68 (5-

300)• Total Days to Publish: 221 (21-

533)• Publication Fees: $2000 • Thousands of dollars and hundreds of hours

per paper• Long delays impair every form of assessment• Variance hinders planning research programs

and careers

Peer Review: Outline

• History and evolution– Invention– Current form

• Modern purpose

• Improving peer review

• Post-publication

Original Purpose

Henry Oldenburg, 1665Medical Essays and Observations, 1731

• Reasons for publishing– Disseminating results– Claiming ownership

of results• Reasons for peer review– Establishing veracity

of results– Deciding which results

to publish given limited space

Modern Purpose• Reasons for

publishing– Disseminating results– Claiming ownership

of results• Reasons for peer review– Establishing veracity

of results– Deciding which results

to publish given limited space

Abstract: 9495Full Text: 8564PDF: 2549

Pre-reception

• Peer review allows authors to test the paper before sending it to the whole field– Catches fundamental errors– Highlights important overlooked

issues

Problems with Pre-reception

• Reviews are impure measures of quality

Problems with Pre-reception

• Reviews are impure measures of quality

• Review process is opaque to the wider field

Prioritization of the Literature

• Loop ranks papers based on which journal publishes them– Provides a way of deciding which papers to

read– Compensates for long publication lags

Problems with Prioritization

• Reviews not focused solely on quality• Inefficient (avg. lag is 221 days)• Creates detrimental short-term

incentives

Problems with Prioritization

http://pmretract.heroku.com

Budget per Paper ($1000)

Problems with Prioritization

http://pmretract.heroku.com

Budget per Paper ($1000)Retractions per 100k Papers

Problems with Prioritization

Fang & Casadevall, 2011

Problems with Prioritization

• Reviews not focused on solely on quality

• Inefficient (avg. lag is 221 days)• Creates detrimental short-term

incentives• Doesn’t work

Problems with Prioritization

Problems with Prioritization

Problems with Prioritization

Problems with Prioritization

Peer Review: Outline• History and evolution

– Invention– Current form

• Modern purpose– Pre-reception– Prioritization

• Improving peer review

• Post-publication

Improving the current system• Make publishing more efficient and less

variable• Days to Publish: 221 (21-

533)• Years to First Cite: ~1.5 +

research

• Capture reviews in the prioritization• Total Number of Reviews: 6.3• Total Hours of Reviewing: ~31.5

• Focus reviews on science rather than publication

Decouple peer review from publishing

Proposed System

Decoupling peer review from publishing

• Allows reviewers to focus on scientific merit rather than publication

• Allows reviews to be captured for prioritization and rewards for reviewers

• Makes publishing simpler and more efficient

• Eliminates the short-term incentive to produce least publishable units

Peer Review: Outline• History and evolution

– Invention– Current form

• Modern purpose– Pre-reception– Prioritization

• Improving peer review– Decoupling from publishing

• Post-publication

Format for papers

The need for post-publication review

Field Altering

Useful Datapoint

Why?

Shrug

Pro

port

ion

of

field

Field Altering

Useful Datapoint

Why?

Shrug

The need for post-publication review

We need more reviews to approximate these complex distributions.

Pro

port

ion

of

field Also enables

personalized prioritization of the literature.

Incentivizing post-publication review

Organizing the

literature

• The utility of modern peer review and publishing is prioritizing and organizing the literature.

Organizing the

literature

• The utility of modern peer review and publishing is prioritizing and organizing the literature.

Peer Review: Outline• History and evolution

– Invention– Current form

• Modern purpose– Pre-reception– Prioritization

• Improving peer review– Decoupling from publishing

• Post-publication review??????