Responsible metrics for research - Jisc Digifest 2016
6
HEFCE Metrics Review 2015: Main Recommendations Responsible metrics • Mind your language: indicators, not metrics • Metrics inform but do not replace peer review • Institutions need to be transparent about use • Clearly state of principles for assessment • Do not delegate measures of excellence to league tables or journals • Dialogue with staff • Data should to be transparent (challenge to providers) • Build on DORA/Leiden Manifesto Prof. Stephen Curry Imperial College London
Transcript of Responsible metrics for research - Jisc Digifest 2016
HEFCE Metrics Review 2015: Main Recommendations
Responsible metrics• Mind your language: indicators, not metrics• Metrics inform but do not replace peer review• Institutions need to be transparent about use
• Clearly state of principles for assessment• Do not delegate measures of excellence to league tables or journals• Dialogue with staff
• Data should to be transparent (challenge to providers)• Build on DORA/Leiden Manifesto
Prof. Stephen CurryImperial College London
Words are not enough: metrics are a deep cultural problem (reified by the REF)
“In some cases the culture of scientific research does not support or encourage scientists’ goals and the activities they believe to be important for… high quality science."
http://nuffieldbioethics.org/project/research-culture/
“The most common complaint from reviewers is that authors are overselling their work.”
Jan 2015
Aug 2014
Aug 2012
0-4
5-9
10-14
15-19
20-24
25-29
30-34
35-39
40-44
45-49
50-54
55-59
60-64
65-69
70-74
75-79
80-84
85-89
90-94
95-99
100-104
105-109
110-114
115-119
120-124
125-129
130-134
135-139
140-144
145-149
150-154
155-159
160-164
165-169
170-174
175-179
180-184
185-189
190-194
195-199
200-204
205-209
210-214
215-219
220-224
0
5
10
15
20
Citations per paper in 2010
% o
f Pap
ers
Data for Nature Materials, 2010
Impact factor = 29.897
Mode = 7
Range (>2 orders of magnitude)
Distribution highly skewed:Highest cited 15% of papers account for 50% of citationsTwo-thirds of papers perform less well than the JIF
http:
//w
ww
.nat
ure.
com
/doi
finde
r/10
.103
8/
Evidence: JIFs mask real variation in citation performance
Kravitz, D. J., & Baker, C. I. (2011). Frontiers Comp. Neurosci. doi: 10.3389/fncom.2011.00055
Distribution highly skewed:Highest cited 15% of papers account for 50% of citationsTwo-thirds of papers perform less well than the JIF
Evidence: Distributions are similar: all journals have highly & lowly cited papers
See also: https://quantixed.wordpress.com/2016/01/05/the-great-curve-ii-citation-distributions-and-reverse-engineering-the-jif/
All journals should publish their citation distributions
Evidence: Correlations between JIF and article citations for individual scientists are poor
“…authors do not necessarily publish their most citable work in journals of the highest impact, nor do their articles necessarily match the impact of the journals they appear in.”
Seglen, P. O. (1997). BMJ, 314, 498–502.
r=0.05 r=0.27
r=0.63r=0.44
Journal Impact Factor
No o
f cita
tions
per
arti
cle
A B
DC
Data may not be enough: re-define good behaviour and incentivise it
Ron Vale (2012) Mol. Biol. Cell, 23, 3286-3289
“As stewards of our profession, academic scientists have a collective responsibility to consider how to disseminate knowledge through publications and how to advance (careers)…”
• Research quality and reproducibility (not just novelty)
• Publishing openly and efficiently (preprints, open access, sharing data & reagents)
• Mentoring and citizenship (e.g. contributions to peer review)
Do we still remember what universities are for?
