MEASURING RESEARCH IMPACT ON THE WEB

Charleston Conference, November 2013

Iain HrynaszkiewiczOutreach Director, Faculty of 1000

iain.hrynaszkiewicz@f1000.comhttp://f1000.com

@iainh_z

The Seer of Science PublishingScience 4 October 2013: Vol. 342 no. 6154 pp. 66-67 DOI: 10.1126/science.342.6154.66 http://www.sciencemag.org/content/342/6154/66.full.pdf

http://blog.f1000.com/2013/10/04/vitek-tracz-science-interview/

1. Individual papers and scientists judged on journal-based metrics

2. Closed, pre-publication peer review3. Lack of access to original research – and data4. Lack of credit for many of scientists’ contributions

SOME PROBLEMS WITH SCHOLARLY COMMUNICATION

SOME PROBLEMS WITH SCHOLARLY COMMUNICATION #1

“[Journal Impact Factor is] a poor indicator of citations to specific

papers or of the future performance of individual researchers”

Nature Materials 12, 89 (2013)

“I am sick of impact factors and so is science.”

Prof Stephen Curry, Imperial College, Aug 2012

http://occamstypewriter.org/scurry/2012/08/13/sick-of-impact-factors/

“The widely held notion that high-impact publications determine who gets academic

jobs, grants and tenure is wrong.”

Dr Michael Eisen, UC Berkeley, Feb 2012http://www.michaeleisen.org/blog/?p=911

“Citations are heavily gamed and are painfully slow to accumulate, and overlook increasingly important societal and clinical

impacts.”

Priem et al., PLoS ONE 7(11): e48753

http://am.ascb.org/dora/

Themes of DORA recommendations:

1.Eliminate the use of journal-based metrics in funding, appointment, and promotion considerations2.Assess research on its own merits not the journal3.Capitalize on the opportunities provided by online publication such as exploring new indicators of significance and impact

DORA has been signed by >9000 individuals and nearly 400 organisations. Original signatories included American Association for the Advancement of Science (AAAS), American Society for Cell Biology, Howard Hughes Medical Institute, Faculty of 1000, Public Library of Science

IMPACT FACTORS AND CITATION METRICS

Advantages Disadvantages

Reproducible Slow – delay of up to 2 years

Transparent calculation (kind of) Data not publicly available

Curated – some human filtering Can be manipulated

Predictor of journal quality Poor predictor of paper and researcher quality

I, Cawi 2001 [CC-BY-SA-2.5 (http://creativecommons.org/licenses/by-sa/2.5)], via Wikimedia Commonshttp://commons.wikimedia.org/wiki/File:Internet-Sign.jpg

“ALTERNATIVE” METRICS

Source: Wikipedia, http://en.wikipedia.org/wiki/Alternative

ALTMETRICS – WHY ARE THEY IMPORTANT?

“For all new grant applications from 14 January, the US National Science Foundation (NSF) asks a principal investigator to list his or her research

“products” rather than “publications” in the biographical sketch section. This means that, according to the NSF, a scientist's worth is not dependent

solely on publications. Data sets, software and other non-traditional research products will count too.”

Heather Piwowar, UBC, in Nature Jan 2013http://www.nature.com/nature/journal/v493/n7431/full/493159a.html

ALTMETRICS – WHY ARE THEY IMPORTANT?

“Metrics --- such as the number of article citations, your h-index or others (such as those available at ImpactStory.org) --- can be useful in making the

case that the publication or scholarly work was significant. The Faculty Promotions Committee discourages the use of journal-based metrics (such

as journal impact factors), since it is the quality and importance of the research contribution itself that is the key.”

Excerpt from UC Denver promotions handbook guide to dossier preparation

TOOLS FOR SHARING RESEARCH PRODUCTS...

AND TOOLS TO INTERACT WITH THEM

Source: http://impactstory.org/collection/v7b25c

Source: http://www.altmetric.com/details.php?citation_id=938211

(ALT)METRICS – ADVANTAGES AND DISADVANTAGES

Advantages Disadvantages

Fast – data available immediately Heterogeneity across tools

Transparent (mostly) Can lack context/meaning

Lots of open data and tools available Can be manipulated

All research products tracked Not always reproducible (ephemeral)

Much broader picture of impact Emergence and disappearance of tools

WHAT’S OFTEN MISSING?

By Taco Hoekwater; Wolfgang Schuster; Xan; svg from Lumu [CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commonshttp://upload.wikimedia.org/wikipedia/commons/0/0d/ConTeXt_Unofficial_Logo.svg

HOW F1000PRIME WORKS

• Faculty includes over 5000 peer-nominated scientists and clinical researchers and ~5000 Associates

• Faculty Members select, rate and comment on the most interesting and important research articles (2-3% of the life science literature) from ~3,700 journals

• Assigns one of three positive ratings: Exceptional (3 stars), Very Good (2 stars) or Good (1 star)

• Text also serves as a short, expert, plain English recommendation written for a global readership

• Adds relevant classifications (e.g. changes clinical practice)

• Publishes about 1500 recommendations per month (>143,000 published to date)

EXAMPLE RECOMMENDATION

F1000, ALTMETRICS AND RESEARCH ASSESSMENT/IMPACT

The Wellcome Trust published in PLOS ONE an analysis of F1000Prime, finding that expert review highlights important papers that bibliometric indicators alone would miss (http://bit.ly/F1000Wellcome).

A 2009 study by the MRC concluded that F1000Prime recommendations are an indicator of future citation impact (http://bit.ly/F1000future).

Collection of studies of F1000Prime:

http://www.mendeley.com/groups/3748101/studies-involving-faculty-of-1000-s-f1000prime/

Also, studies have shown that tweets and number of readers on Mendeley predict citations.

How much does all this matter?

SOME PERSPECTIVE. WHAT IS IMPACT?

IS THIS IMPACT?

Source: http://www.jmedicalcasereports.com/mostviewed/alltime

THIS IS IMPACT

Source: http://www.ncbi.nlm.nih.gov/pubmed/15555666

SUMMING UP

• Impact factors are not good for assessing individual papers and individuals

• The internet has given us a wealth of new data and tools to give a broader picture of the impact of research

• We are still trying to understand the meaning of many new metrics – we need data with context

• Some “alternative” metrics are beginning to influence research assessment

• All metrics – citation and non-citation metrics – have advantages and disadvantages