Measuring science with meaningful metrics Interdisciplinarity and transdisciplinarity Edwin Horlings...

Measuring science with meaningful metrics

Interdisciplinarity and transdisciplinarityEdwin Horlings

23 januari 2014

2 | Kenniscoproductie voor de grote maatschappelijke uitdagingen

Meaningful metrics for evaluation

• Why do you evaluate?• accountability• learning

• What do you evaluate?• output• impact• process

• What metrics are meaningful?

MONODISCIPLINARY SCIENTOMETRICS


How does a scientist develop his/her portfolio?

• Empirical evidence for the reward system of science• priority seeking: strive to be the first to solve a problem• receive recognition from a community of peers

• Search strategies• Search: the process by which an individual scientist identifies, enters, develops, and exits a problem area and its associated community of peers.• Strategy: the scientist’s strategic positioning in a competitive environment presumes a degree of planning, coherence and consistency to problem choice over time.

Horlings, E., & Gurney, T. (2013). Search strategies along the academic lifecycle. Scientometrics, 94(3), 1137-1160.


The portfolio of Ronald Plasterk

Bijzonderhoogleraar

moleculaire microbiologieVrije Universiteit

1993-2003

DNA inversions by Gin protein inbacteriophase Mu and escherichia-coli

Tc1 and Tc3 transposase of C-elegans

Reverse genetics andgene mapping in Caenorhabditis elegans

HIV integrase protein

Transposon silencing and RNA interferencein Caenorhabditis elegans

Target-selected mutagenesisin zebrafish and rats

MicroRNA expression

tijd

prob

leem

gebi

eden

doctoraalbiologie

1981

PhD1984

NKIOnderzoeksschool

Oncologie1987-2000

Directeur Hubrecht InstituutHoogleraar ontwikkelingsgenetica

Universiteit Utrecht2000-2007


Six propositions

1. A scientist’s work consists of multiple finite research trails2. A scientist will work in several parallel research trails3. A scientist’s role in research trail selection changes along

the lifecycle4. The start and end of research trails is associated with career

changes5. The start and end of research trails is associated with the

potential for reputational gain6. A scientist’s portfolio will converge before it diverges



Proposition 2: Parallel research trails

0

1

2

3

4

5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

Years since first documented publication

Num

ber o

f par

alle

l res

earc

h tr

ails

PhD

Professor

Postdoc


Proposition 3: Role in problem selection changes along the lifecycle

Percentage of publications written in first, other or last author positions in three phases of the academic lifecycleauthor position PhD Postdoc Professorfirst 56.4 22.8 19.7other 28.9 39.6 29.4last 14.8 37.6 50.8

total 100 100 100



Proposition 5: Start and end of trails associated with reputational gain

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30Years of publishing within a problem area

Ave

rage

ann

ual c

itat

ions

per

pap

er

short (five years or less in duration)

short (five years or fewer active)

intermediate

intermediate (excluding outliers)

lifetime

INTER/TRANSDISCIPLINARY SCIENTOMETRICS


Definitions

• Interdisciplinary research (IDR) integrates separate disciplinary data, methods, tools, concepts, and theories in order to create a holistic view or common understanding of a complex issue, question, or problem

• Transdisciplinary research (TDR) integrates knowledge by researchers and stakeholders from different fields, working on a common problem over an extended period of time, developing shared conceptual frameworks, skills, and goals, transcending disciplinary perspectives

Wagner, C.S. et al. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal of Informetrics, 5(1), 14-26.

Rosenfield, P. L. (1992) ‘The potential of transdisciplinary research for sustaining and extending linkages between the health and social-sciences’, Social Science & Medicine, 35 (11), 1343-57.Klein, J. T. (2008) ‘Evaluation of interdisciplinary and transdisciplinary research - A literature review’, American Journal of Preventive Medicine, 35 (2), S116-S23.


Integration is the key

Involves different knowledge producers

disciplines, societal actors

Combines heterogeneous knowledge

scientific, technical, experiental, social

Produces wider variety of outputs

peer-reviewed papers, policy reports, tools, etc.

Different production process creating consensus; aligning incentives; managing expectations; etc.

New performance criteria new notions of impact and quality


IDR: social network analysis

Physicsgood coverage

Sociologypoor coverage

Computer science

conference proceedings


Measuring the characteristics of TDR

• Funding information: co-financing by stakeholders

• Collaboration: involvement of stakeholders

• Output: variety


TDR funding

Acknowledgements and funders were correctly retrieved

Yes NoAuthors

acknowledge KVK programme

funding

Yes 57 (26%) 60 (27%)

No 66 (30%) 20 (9%)

N=221; 18 papers (8%) without acknowledgement in the metadata

Koier, E., & Horlings, E. (2014). “How accurately does output reflect the nature and design of transdisciplinary research programmes?” Research Evaluation, rvu027 (in press).


TDR collaborators

Type of organisation

Articles co-authored by at least one author affiliated to a type of organisation

Percentage of total articles (n=304)

University 251 83%

Public research organisation 193 63%

Government 6 2%

Firm 27 9%

NGO 6 2%



TDR outputs

Output classified by the project leaders as scientificScientific papers, peer reviewed 542Scientific papers, other 106Proceedings 115PhD theses 11

Output classified by the project leaders as non-scientificAudio 2

Books 43

Brochures 68

Final project reports 171

Media 322

Press releases 26

Popular articles about science 117

Posters 159

Presentations 1,087

Project factsheets 247

Project newsletters 81

Reports 515



Impact

• Societal impact is the holy grail of TDR

• Does TDR produce societal impact?

• Survey among project leaders and participants of 178 projects in Climate changes Spatial Planning (2004-2011) and Knowledge for Climate (2008-2014)

Stefan P.L. de Jong et al. (2014). “Exploring the promises of transdisciplinary research: a quantitative study of two climate research programmes” (draft paper)


Different types of impact

• project results:• were used in societal debates• contributed to including climate change knowledge in investment

decisions• created political support for climate adaptation measures• helped to postpone or cancel climate adaptation measures• produced climate adaptation measures and strategies that were

implemented• created new contacts• improved existing contacts


Each impact requires a custom design Societal

debateInvestment decisions

Political support

Implementa-tion

Improved contacts

New contacts

Share of societal actors in the project – Number of informally involved societal actors + + + +Prominence of societal actors in the project + + Influence of societal actors on research process + Number of publication types aimed at non-scientific audience + + + + +


Lessons for the evaluation of TDR

• Multi-method approach of which scientometrics is just one

• Calls for new criteria (dealing with variety of knowledge inputs and outputs)

• Evaluate (project)specific impacts

• Stronger emphasis on process and design, not just output

Thank you for your attention

Edwin Horlings | [email protected]


Risk versus reward in science

rew

ard

riska1

b1

a0

b0

a2

b2

a0 ,b 0

Wouter P.C. Boon & Edwin Horlings (2014), Knowledge co-production in protective spaces””, draft paper

Measuring science with meaningful metrics Interdisciplinarity and transdisciplinarity Edwin Horlings...

Documents

Transcript of Measuring science with meaningful metrics Interdisciplinarity and transdisciplinarity Edwin Horlings...