The need for contextualized

scientometric analysis

Ludo Waltman and Nees Jan van Eck

Centre for Science and Technology Studies

Leiden University

STI Conference 2016, Valencia, Spain

September 16, 2016

Evaluative scientometrics:

Basic research at CWTS

1

Evaluative scientometrics:

Applied work at CWTS

2

To what degree does research evaluation

truly benefit from our work?

3

Do we fully exploit the possibilities of

scientometric analyses?

What should be the main focus of

research in evaluative scientometrics?

Trends in evaluative scientometrics

• Increasingly sophisticated indicators

• Do-it-yourself bibliometrics

• Informed peer review

• Altmetrics

4

Simple vs. advanced indicators

Simple indicators

• Low accuracy

• Easy to understand

• Limitations are evident

Advanced indicators

• High accuracy

• Hard to understand

• Limitations are hidden in

technical details

5

Dilemma of informed peer review

Domain experts

• Deep domain knowledge

• No scientometric

knowledge

Scientometricians

• Deep scientometric

knowledge

• No domain knowledge

6

How should these two groups communicate?

A common language is lacking

Dilemma of informed peer review

• Triangulation of expert knowledge and scientometric

indicators is difficult without a common language

• Scientometric indicators are either rejected or blindly

trusted

• Overemphasis on what can be quantified easily

7

Contextualized scientometric analysis

• Context

– Allow data underlying an indicator to be explored

• Simplicity

– Increased sophistication comes at a price; value simplicity

– Stop being overly negative about impact factors and h-indices

• Diversity

– To avoid overemphasizing what can be quantified easily, diversity of

indicators is essential

– Better to have indicators of low accuracy than no indicators at all

8

9

10

11

Advanced bibliometric indicators vs.

contextualized scientometric analysisAdvanced bibliometric

indicators

• Offer reasonably accurate

but incomplete

information

• Little room for deeper

understanding

• To be used for macro-

level evaluations

Contextualized

scientometric analysis

• Offers detailed

information requiring

careful interpretation

• Allows triangulating

expert knowledge and

indicators

• To be used for micro-level

and meso-level

evaluations

12

Conclusions

• Scientometricians should shift their attention from

advanced bibliometric indicators to contextualized

scientometric analysis

• New research questions:

– How to make contextual information available (e.g., interactive user

interfaces, visualizations)?

– How to guide domain experts in the use of this information?

– How to fulfill the promise of informed peer review?

13

Thank you for your attention!

14