Research evaluation: is it our business? Librarians in the brave new world of research evaluation

Andria McGrathSenior Information Specialist, Research Support, King’s College London

What is research evaluation?

• External evaluation – in the UK the RAE now REF3 major elements of the RAE: – publication outputs, – grant income awarded, – research students

• Vital importance – link to income• Benchmarking – Rankings• Internal evaluation

The players in institutions

• Research Office• PVC Research• Research group/dept heads• Administrators• Librarians – publications expertise• Institutional repository managers

3

Evolving interest

• REF events at King’s 2008 -2010• 1st event – Beyond the RAE – representatives

from 75 universities – 1/3 librarians• Survey –

– biggest challenges – verifying publication lists– Biggest concerns for REF – bibliometrics – data

sources, subject differences, skilling up to cope

• http://www.kcl.ac.uk/iss/support/ref

4

Events Nov 08 and June 09

• Focus on bibliometrics – pilots• Pre-event survey before Nov event:

– Only about half of institutions had a centralized publications data collection system

– One or two mentioned Symplectic– One mentioned something called Pure

5

Data collection systems

• Some – only concerned with collecting RAE pubs• Others – aimed at collecting all publication info• Wider uses – profile pages / internal evaluation• Some creating internal ‘home grown’ systems

– Separate systems for 3 RAE data elements– Integrated system, taking feeds from other systems

(eg Research Gateway at King’s)

REF focus changed

• Bibliometrics element toned down• Impact (4th King’s REF event in June 2010)• Not publication impact – practical impacts of

research• Research leaders had seen the light on

– data collection systems – bibliometrics

7

CRIS Current Research Information SystemsAlready common in mainland Europe

8

CERIF

• Common European Research Information Format

• A standard – uses XML• Objects or entities with attributes

eg project, person, organizational unit; • Relationships – 'linking relations’ • Rich semantics – roles and time

9

R4R project - Ready for REF

• JISC project – King’s and Southampton• CERIF4REF XML schema created• Plugins for Eprints, Dspace and Fedora• Transfer data to REF collection system• Research admins, IR managers, systems staff

and librarians involved• http://r4r.cerch.kcl.ac.uk/

10

Librarians and CRIS systems

• St Andrews experience• If research office led – get in on the ground

floor• Librarians have plenty of expertise to offer

11

Which librarians?

• Senior managers• Subject/liaison librarians• Research support librarians• Instutional repository managers• Information resources depts. – subs• Enquiries staff

12

Strategic directions for libraries

• RLUK strategic themes• 1st – redefining the research library model• New services / new roles for staff• Value of libraries for research and researchers

(RIN/RLUK report)• RLUK – Reskilling for Research (forthcoming)• Research support librarians

13

Some new areas for librariansassociated with research evaluation• CRIS systems & CERIF• New ways of interacting with bibliographic /

citation databases• APIs – for publication or citation data• Discovering other institutional systems

eg grants DBs / student systems that feed into CRISs

• Bibliometrics principles and new products14

New opportunities

• Showing our value– To administrators / research office– To academics and senior research leaders– To Graduate School

• Opportunities to advocate Open access

15

Bibliometrics – the new buzz word

Awareness growing

•By librarians

•By research leaders in institutions

•By academics

16

Librarians – natural synergy

• Citation databases• Citation reports• Training offered – eg by Thomson Reuters• Training points up limitations of basic

measures• Normalizing of measures crucial but difficult

without specialised products

17

By research leaders

• A few initiates who understand the issues leading the way

• Percentiles tables in Essential Science Indicators used for evaluations

• H-index being used by some• At least one institution has employed a

bibliometrician

18

Academics

Some beginning to take an interest

•Discovering:– ResearcherID– Citation reports in citation databases– Google Scholar

•Graduates – training opportunity – “Making an impact”

19

Holy grail

• Research information system• Dashboard for research managers – eg for

grant information• Captures data with minimum effort• Integrated with bibliometrics• For internal & external purposes – evaluation

and promotional

20

Is it our business?

21