Dalian 21-23 May 2012

The role of peer review in the selection of academic talentPleun van Arensbergen

21-23 May 2012

Pleun van Arensbergen | 2 |

Variation in peer review

- Peer review is used for different purposes, involving various objects, e.g. scientific papers, research proposals, research programs, grant applications.

- Considered to be ‘the’ system of quality control for and within the scientific community, although contested

- Also used for selection of academic talents

21-23 May 2012

But does it work?

Previous research on applications for personal grants shows that peer review scores do not predict application success (Van den Besselaar & Leydesdorff 2009)

What is the role of peer review in talent selection?

Quantitative study of the review procedure of 905 grant applications from 2009 Innovational Research Incentives Scheme

Pleun van Arensbergen | 3

21-23 May 2012

Dutch Personal Grant Program

- Personal grants for excellent researchers belonging to the top 10-20% of their population, in three career phases

• Early career (max €250.000)• Intermediate career (max €800.00)• Advanced career (max €1.500.000)

- Growing importance of funding program- Increasing number applications, decreasing allocation rate- Training and preselection at universities

Pleun van Arensbergen | 4 |

21-23 May 2012

Selection procedure

Panel criteria

-Quality of applicant-Quality of proposal-Research impact

1540 No data

905 635 (41%)

552 353 (39%)

263 289 (52%)17 % granted

21-23 May 2012

Talent identified by peers?

- No clear identification of talent

Pleun van Arensbergen | 6

21-23 May 2012

Role of peer review in selection (1)

- Moderately strong correlation between peer review and panel review (ԏ=.52)

- Strongest correlation with panel review of proposal (ԏ=.55 / .64)

- Weakest correlation with panel review of research impact (ԏ=.22 / .36)

- If selection would have been based on peer review: 17% of current interview candidates would not have been selected for interview, 24% not eventually granted

Pleun van Arensbergen | 7

21-23 May 2012

Predicting talent selection

Model constructed with logistic regression analysis, including

•Only constant: 54% correctly predicted

•Peer reviews: 64% correctly predicted

•All reviews: 83% correctly predicted

• Peer reviews and research impact not included• Only panel review of proposal (B= 1.86) & researcher included (B= 1.40)

Minor role of peer review in selection of talent:

not the experts on topic decide, but panelPleun van Arensbergen | 8

21-23 May 2012

Peer versus Panel

• Peer review as part of selection process• First (large) selection made without peer review!

• Peer review versus panel review:• Peer review: experts on the topic individually review one application,

send back their review• Panel review: good researchers in the discipline review all or a set of

applications, discuss their review within the panel and decide on a ranking

• Level of ‘peerness’

Pleun van Arensbergen | 9

21-23 May 2012

Panel selects

(Based on 30 interviews with panel members)

•Excellent CV (e.g. publications, awards, grants, international experience)

• Can be reviewed for everyone, also by non-peers

•Do I understand the proposal?• Should be understandable for everyone, also for non-peers

•Is it innovative and feasible?• Using (and reviewing) external peer reviews

•Is it their own research idea / line?• Checked during the interview

•Motivation, passion and enthusiasm• Checked during interview

Pleun van Arensbergen | 10

21-23 May 2012

Importance of interview

Pleun van Arensbergen | 11

- clarification - own idea- motivation & enthusiasm

But also- communication skills- personal attitude- having a ‘click’

Many talented researchers, so selection partly dependent on coincidence, but

“real talents are easy to identify”

21-23 May 2012

Talent identification in ‘real life’

Which talents do we want to select for our own department?

-Good CV

-Motivation & enthusiasm

-Fit within the group

- Expertise- Social characteristics (e.g. nice and helpful person, a ‘good

colleague’)

Often talents are not recognized directly, but after working with them for some timePleun van Arensbergen | 12

21-23 May 2012

Recommendations

• Provide peers with a format for their review, including a clear motivation of their argument

• Maybe only for evaluation of innovativeness and feasibility

• Let the panel only identify the top and bottom, the middle part will be randomly allocated

• Top applicants are funded directly• Applicants in grey zone are randomly selected

• Do not standardize but use a variety of selection procedures

Pleun van Arensbergen | 13

21-23 May 2012

p.vanarensbergen@rathenau.nl

21-23 May 2012

Predicting interview selection

Model constructed with logistic regression analysis, including

•Only constant: 60% correctly predicted

•Peer reviews: 69% correctly predicted

•All reviews: 77% correctly predicted

• Peer reviews and research impact not included• Only panel review of proposal (B= 1.36) & researcher included (B= 0.71)

Pleun van Arensbergen | 15