OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn -...

35
Next-generation Metrics for Scholarly Performance William Gunn, Ph.D. Head of Academic Outreach Mendeley @mrgunn https://orcid.org/0000-0002-3555-2054

description

Presentation at the OpenAIRE-COAR Conference: "Open Access Movement to Reality: Putting the Pieces Together", Athens - May 21-22, 2014. Session 4: The impact of openness and how to evaluate research. Next generation metrics of scholarly performance, by William Gunn - Head of Academic Outreach for Mendeley

Transcript of OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn -...

Page 1: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Next-generation Metrics for Scholarly Performance

William Gunn, Ph.D. Head of Academic Outreach

Mendeley

@mrgunn

https://orcid.org/0000-0002-3555-2054

Page 2: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Open Access Policies

Page 3: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Open Access Has Benefits

• The perception that there were tradeoffs when choosing open access slowed adoption

• Only Open Access can fully provide the benefits of the web – interoperability, accessibility, discovery

• An expanded definition of impact brings new kinds of content and new metrics

Page 4: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

King, Christopher (2012) Thomson Reuters Annual Report

http://ar.thomsonreuters.com/_files/pdf/MultiauthorPapers_ChrisKing.pdf

Page 5: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Adams, Jonathan. "Collaborations: the fourth age of research."

Nature 497.7451 (2013): 557-560.

Page 6: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Amgen: 47 of 53 “landmark” oncology publications could not be reproduced

Bayer: 43 of 67 oncology & cardiovascular projects were based on contradictory results

Dr. John Ioannidis: 432 publications purporting sex differences in hypertension, multiple sclerosis, or lung cancer. Only one data set was reproducible

There is no gold standard

Page 7: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph

publishing." new media & society 15.3 (2013): 359-378.

Authors Publishers

Printers

Shippers

Booksellers

Readers

Page 8: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph

publishing." new media & society 15.3 (2013): 359-378.

Authors Publishers

Printers

Shippers

Booksellers

Readers

1770

Page 9: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Authors Readers Internet

Research is changing

Publishers

Repositories

Professional

Websites

Page 10: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Challenges for Repositories

• The world is leaning green

• Repositories thus have an essential role

• Repositories aren’t in the workflow of a researcher, neither in search for new ideas nor in presentation of their work

Page 11: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

The Ouroboros problem

• repositories need content deposited to get usage, and they need usage to get deposits

• Discovery tools can address usage and usage will promote content

– scholarly outputs are papers, datasets, code, and more.

Page 12: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

New forms of discovery

• personalized recommendations based on reading history

• related articles

• recommender frameworks

–implement recommendations as a service

• third-party recommender services

–serve niche audiences

Page 13: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

research impacts more than authors

http://altmetrics.org/manifesto/

Page 14: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

...and aggregates research data in the cloud

Mendeley extracts research data…

Install Mendeley Desktop

Collecting rich signals from domain experts.

Page 15: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Home Work Mobile

Cloud Library

Page 16: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Shared Folder

Page 17: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Mendeley Research Catalog

Page 18: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Mendeley Research Catalog

Page 19: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Read papers + keep track of notes

470M documents

Page 20: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Taking some misery out of writing

Page 21: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Information Extraction

Page 22: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

We are publishing this data to the LOD cloud

http://code-research.eu/

Page 23: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Defining readership

• Each document addition is a “read”

• stamped with metadata describing the context of the read event

• a read is like a citation, but faster and captures more

Page 24: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

research impacts more than authors

http://altmetrics.org/manifesto/

Page 25: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley
Page 26: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley
Page 27: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Readership vs. citations

• it comes with a payload of metadata

• it accrues faster

• it illuminates previously hidden impact

Page 28: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

http

://arx

iv.org

/htm

l/1203.4

745v1

altmetrics show broader impact

Page 29: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

htt

p://d

x.d

oi.org

/10.3

789/isqv25no2.2

01

3.0

4

Page 30: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

http://www.asis.org/Bulletin/Apr-13/AprMay13_Lin_Fenner.html

altmetrics show broader impact

Page 31: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Issues To Be Addressed

• Identity

• Privacy

• Attribution

• Gaming

• Filtration

standards/ best practice

Page 32: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Amgen: 47 of 53 “landmark” oncology publications could not be reproduced

Bayer: 43 of 67 oncology & cardiovascular projects were based on contradictory results

Dr. John Ioannidis: 432 publications purporting sex differences in hypertension, multiple sclerosis, or lung cancer. Only one data set was reproducible

There is no gold standard

Page 33: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

Cultural skew is important

South America is weak on N.A. social media, strong on Mendeley

Page 34: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

What would people build if they could get the data?

• Impact Story – get credit for all your work

• PLOS ALM – article-level metrics for papers

• Plum Analytics – bespoke analytics for libraries (EBSCO)

• Altmetric.com – altmetrics for publishers. (Digital Science)

Page 35: OpenAIRE-COAR conference 2014: Next generation metrics of scholarly performance, by William Gunn - Mendeley

www.mendeley.com

[email protected] @mrgunn