Academic Publishing is Evolving… How Should We ‘Evaluate’ Scientific Publications Today? Pete...

Academic Publishing is Evolving…

How Should We ‘Evaluate’Scientific Publications Today?

Pete Binfield Co-Founder and Publisher PeerJ

Samuel Merritt - 10/30/2013

@ThePeerJhttps://peerj.com

@[email protected]


• What do we mean when we say ‘Evaluate’?• Evaluating ‘Impact’ or ‘Reception’ or ‘Reach’ or ‘?’• Providing Subjective Opinions & Evaluations• Evaluating ‘Integrity’

• And at what level of granularity?• The journal?• The article?• The paragraph?


#1. Evaluating ‘Impact’ or ‘Reception’ or ‘Reach’ or ‘Interest’

or ‘Readership’ or, or, or


• Online-only, peer-reviewed, open access journals• covering a very broad subject area• selecting content based only on ‘technical

soundness’ (or similar) • with a business model which allows each article to

cover its own costs

Open Access ‘MegaJournals’


Q2 13

Q1 13

Q4 12

Q3 12

Q2 12

Q1 12

Q4 11

Q3 11

Q2 11

Q1 11

Q4 10

Q3 10

Q2 10

Q1 10

Q4 09

Q3 09

Q2 09

Q1 09

Q4 08

Q3 08

Q2 08

Q1 08

Q4 07

Q3 07

Q2 07

Q1 07

Q4 06

0

1000

2000

3000

4000

5000

6000

7000

8000

Year Pubs Notes2007

1,200 Larger than ~ 95% of all journals

2008

2,800 Largest OA journal in world

2009

4,400 3rd largest journal in world

2010

6,750 Largest journal in world

2011

13,800 ~1.4% of PubMed output in that year

2012

23,500 ~2.4% of PubMed output in that year

2013

~31,000

>3% of the literature

PLOS ONE Quarterly Output


Known MegaJournals (Oct 2013)


In addition, if we allow for narrow scope ‘megajournals’ then we should also include:

• All of the “Frontiers in…” Series (part of Nature)

• All of the “BMC Series” (~ half of BMC)

• ~ 1/3 of Hindawi’s current output

All of these titles refuse to pre-judge what the audience should be reading (other than determining that the content should join the literature).


An OA future containing MegaJournals

PLoSONE

SAGEOpen

PeerJ

ALL

OTHER

OA

JOURNALS

etc.etc.


The Effect of the ‘MegaJournal’

• Rapidly Approaching ~10% of all published content, spurring new developments

• Require, and have given rise to, Article-Level Metrics

• Publish Negative Results, Replication Studies, Incremental Articles

• Dramatic Improvement to the Speed of the Ecosystem

• Dramatic Improvement to the Efficiency of the Ecosystem

Academic Publishing is Evolving…From “Article- ‐Level Metrics, A SPARC Primer” - http://www.sparc.arl.org/sites/default/files/sparc-alm-primer.pdf

http://www.sparc.arl.org/sites/default/files/sparc-alm-primer.pdf

http://www.sparc.arl.org/sites/default/files/sparc-alm-primer.pdf

Academic Publishing is Evolving…Screenshot from ~ Nov 2009 but Way Back Machine has examples from April 2008


PLOS ALMs


PeerJ ALMs

Academic Publishing is Evolving…http://www.altmetric.com/details.php?doi=10.7717/peerj.182

http://www.altmetric.com/details.php?doi=10.7717/peerj.182

http://www.altmetric.com/details.php?doi=10.7717/peerj.182


• Rapidly Approaching ~10% of all published content, spurring new developments

• Require (and have stimulated) Article-Level Metrics

• Publish Negative Results, Replication Studies, Incremental Articles

• Dramatic Improvement to the Speed of the Ecosystem

• Dramatic Improvement to the efficiency of the way the ecosystem currently ‘filters’ content

The Effect of the ‘MegaJournal’


“rejected from at least six journals (including Nature, Nature Genetics, Nature Methods, Science) and took a year to publish before going on to be my most cited research paper (150 last time I looked)” – Cameron Neylon

Academic Publishing is Evolving…http://grigoriefflab.janelia.org/rejections


http://blog.rubriq.com/2013/06/03/how-we-found-15-million-hours-of-lost-time/

“…in a recent report Kassab and his colleagues estimated that Elsevier currently rejects 700,000 out of 1 million articles each year.”

http://poynder.blogspot.co.uk/2013/10/media-research-analyst-at-exane-bnp.html


#2. ‘Subjective’ Opinions & Evaluations(i.e. contextual, human evaluations)


PeerJ Q&A


PMC Commons


#3. Evaluating ‘Integrity’


Open Peer Review would makethis problem disappear.

Overnight


• Atmospheric Chemistry and Physics - Reviewers comments published on pre-pub discussion site. Reviewer names optional.

• Biology Direct - Reviewer comments published, and reviewers named

• BMJ Open - All reviewers named, all reports public• eLife - Decision letter published with articles with author

approval. Reviewers anonymous, but editor named. • EMBO journal - Review process file published with

articles. Reviewers anonymous, editor named. • F1000Research - All reviewers named, all reports

public. • Frontiers journals - Reviewers named, but reports not

public• GigaScience - Pre-publication history published with

articles, and reviewers named. (encouraged, opt-out)• Medical journals in the BMC series - Pre-publication

history published with articles, and reviewers named (encouraged).

• PeerJ - Peer review history published with articles with author approval. Reviewers encouraged to sign report.

Journals Practicing Open Peer-Review


• ~40% of PeerJ Reviewers name themselves.• ~80% of PeerJ Authors reproduce their peer review history


• What do we mean when we say ‘Evaluate’?• Evaluating ‘Impact’ or ‘Reception’ or…• Providing ‘Subjective’ Opinions & Evaluations• Evaluating ‘Integrity’

• And at what level of granularity?• The journal?• The article?• The paragraph?


Thank You

Pete BinfieldCo-Founder and Publisher

@[email protected] @ThePeerJ

Academic Publishing is Evolving… How Should We ‘Evaluate’ Scientific Publications Today? Pete...

Documents

Transcript of Academic Publishing is Evolving… How Should We ‘Evaluate’ Scientific Publications Today? Pete...