R&D Evaluation Methodology &

Funding Principles

Pilot testing

Conference in Prague, Oct 15, 2015

Vlastimil Růžička and KA4 team

www.metodika.reformy-msmt.cz

Pilot Testing („PT“) of the new

evaluation methodology

• Main objectives:

• comprehensively test the methodology including all its processes

• provide feedback for the future full-fledged evaluation

• Only a very limited number of RO selected, which perform exclusively or

partly research in chemistry and history

• Period for evaluation: 2010-14 (outputs 2009-13)

• Results of PT will not be published; however, results will be submitted to RO

for comments and feedback

• Results of PT will not influence the institutional funding for R&D

• The primary objective is not to evaluate the ROs, but to test the

methodology.1

Assessment criteria and sub-criteria

Assessment criteria Sub-criteria

I Research environment

The quality of the research management

(including HR management)

The adequacy of the research strategy

IIMembership of the global and national

research community

International research presence and

collaboration

National research presence and

collaboration

III Scientific research excellence

IV Overall research performance

Research output (including quantity and

overall quality)

Competitiveness in research

V Relevance for society

2

Pilot testing - timetable

3

XII/2014 I/2015 II/2015 III/2015 IV/2015 V/2015 VI/2015 VII/2015 VIII/2015 IX/2015 X/2015

invitation of ROs to the pilot testing

registration of RUs into the pilot testing

elaboration of the self-assessment report

searching for and contracting panel members

searching for and contracting reviewers

assessment of excellent outputs

calibration meeting of panel members14.,20.,21.5.

remote assessment of RUs

meeting of evaluation panels in Prague 29.6.-10.7.

elaboration of RU evaluation reports

elaboration of EvU evaluation reports

elaboration of panel members’ feedback to Metodika

elaboration of feedback to Metodika by evaluated units (RU, EvU)

List of participating EvUs; NS+ET

4

Name of EvU Type of RO

Nr. RUs in EvU

RUs registered in FoS

NA

TUR

AL

SCIE

NC

ES,

ENG

INEE

RIN

G

AN

D

TEC

HN

OLO

GY

University of Chemistry and Technology Prague - Faculty of Chemical Technology

ScRO-HEI 2 1.4, 2.5

University of Chemistry and Technology Prague - Faculty of Environmental Technology

ScRO-HEI 2 2.4, 2.7

University of Chemistry and Technology Prague - Faculty of Food and Biochemical Technology

ScRO-HEI 3 1.4, 1.6, 2.9

University of Chemistry and Technology Prague - Faculty of Chemical Engineering

ScRO-HEI 2 1.4, 2.4

Brno University of Technology -Faculty of Chemistry ScRO-HEI 4 1.4, 1.6, 2.5, 2.7

J. Heyrovský Institute of Physical Chemistry of the AS CR ScRO-ASCR 1 1.4

The Institute of Chemical Process Fundamentals of the AS CR ScRO-ASCR 4 1.4, 2.4, 2.5, 2.7

Centre for Organic Chemistry Ltd. IBRO- RTO 1 1.4

The Research Institute of Inorganic Chemistry, Inc. IBRO-RTO 1 1.4

List of participating EvUs; HU

5

• 12 RO, 15 EvU, 31 RU took part in pilot testing

• It was recommended to register RUs that correspond to the internal organizational structure of

RO as much as possible

• No interdisciplinary RUs registered even though RO were encouraged to do so

• The rule to register only one RU in the given FoS (to one subject panel) was adhered to

Name of EvU Type of RO

Nr. RUs in EvU

RUs registered in FoS

HU

MA

NIT

IES

The University of Pardubice - Faculty of Arts and Philosophy ScRO-HEI 3 6.1, 6.2, 6.3

The University of South Bohemia in České Budějovice - Faculty of Philosophy

ScRO-HEI 2 6.1, 6.2

The Institute of History of the AS CR ScRO-ASCR 1 6.1

The Institute for Contemporary History of the AS CR ScRO-ASCR 2 6.1, 6.3

The National Technical Museum NatRes 1 6.1

The National Archives NatRes 2 1.4, 6.1

Research Unit - recommendations

• For a registration of a RU a threshold has to be set, e.g. 50 outputs with a

possibility to increase the significance of „large“ outputs (e.g. SSH books)

• No limit to the maximum size of a RU (number of scientists and/or outputs)

• EvU has a possibility to register more than just one RU in a field of science if it

provides required documentation to justify the reasons (high number of

scientists and clear distinctiveness of sub-fields within the FoS)

• RU should clearly correspond to the organizational structure of the represented

EvU

• EvU does not necessarily need to include all scientific staff to the registered RU,

but it is obliged to inform the Panel about the reasons and the number of not

included scientists

6

Selection of panellists

• The process of searching and appointing the main and subject panel members as well as appointing the reviewers was provisional. It had to run much faster than what was required in the proposed cascade process.

• The selection of main and subject panel chairs, members of main and subject panels and the referees was done by KA4 team.

• Databases of experts who participated in analogous assessment of R&D in different countries was used. These databases included experts registered by Technopolis consortium, members of the main and subject panels in UK RAE2008 and UK REF2014 evaluations, experts from the evaluation in Portugal in 2013 organized by the Portuguese Foundation for Science and Technology, experts from the evaluation of the Royal University in Stockholm, experts from the University of Helsinki evaluations, experts used by the MEYS for the evaluation of its programs funding research.

• Some of the panel members were chosen based upon a reference of the evaluated units requested by KA4 team.

• In the last stage, recommendations from Technopolis Amsterdam for the missing experts in some disciplinary panels was used.

• Addressing and contracting potential experts was launched in January 2015 and finished in early June 2015.

7

List of panels and panellists

OECD Name Surname Name Affiliation, Town, Country

1. NATURAL SCIENCES- mainpanel

Thulstrup Erik Roskilde University, Roskilde, Denmark

Němeček Zdeněk Charles University, Prague, CR

Rejholec Václav Consultant for pharmaceutical industry, Prague, CR

1.4 Chemical Sciences

Hapiot Philippe CNRS- University of Rennes 1, Rennes , France

Guillon Daniel CNRS - University of Strassburg, Strassburg, France

Haines Michael Cofree Technology Ltd, Bricklehampton, UK

Heintz Andreas University of Rostock, Rostock, Germany

Kukhar Valery Nat. Academy of Sciences of Ukraine, Kyiv, Ukraine

Rizzi Andreas University of Vienna, Vienna, Austria

1.6 Biological Sciences

Driessen Arnold JM University of Groningen, Groningen, Netherlands

Elska Ganna National Academy of Sciences of Ukraine, Kyiv, Ukraine

Rodger Alison University of Warwick, Warwick, UK

8

• Main panel: chair-international expert, two Czech members representing research

community and users; third Czech member representing funding agency/ministry missing

• Subject panels: all international experts

• Number of subject panels members corresponded to the number of evaluated RU

List of panels and panellists

OECD Name Surname Name Affiliation, Town, Country

2. ENGINEERING and

TECHNOLOGY - main

panel

Seville Jonathan Peter Kyle University of Surrey, Guildford, Surrey, UK

Hanika Jiří Czech Academy of Sciences, Prague, CR

Souček Ivan University of Chemistry and Technology, Prague, CR

2.4 Chemical Engineering

Lapicque François CNRS-ENSIC University of Lorraine, France

Grievink Johan University of Technology, Delft, Netherlands

Ocone Raffaella Heriot-Watt University, Edinburgh, Scotland, UK

2.5 Materials Engineering

de With Gijsbertus Eidhoven University, Eidhoven, Netherlands

Drillon Marc CNRS - University of Strassburg, Strassburg, France

Katgerman Laurens University of Technology, Delft, Netherlands

Salmi Tapio Åbo Akademi, Åbo (Turku), Finland

2.7 Environmental

Engineering

Rulkens Wilhelmus Henricus Wageningen University, Wageningen, Netherlands

Legube Bernard University of Poitiers, Poitiers, France

Sánchez Hervás José María Unit for Energy Valor. of Fuels and Wastes, Madrid, Spain

2.9 Industrial

Biotechnology

Voragen Fons Wageningen UR University, Wageningen, Netherlands

Jelen Henryk Poznań University of Life Sciences, Poznań, Poland

9

List of panels and panellists

OECD Name Surname Name Affiliation, Town, Country

6. HUMANITIES - main

panel

North Michael University of Greifswald, Greifswald, Germany

Ledvinka Václav Prague City Archives, Prague, CR

Pešek Jiří Charles University, Prague, CR

6.1 History and

archaeology

Hadler Frank University Leipzig, Leipzig, Germany

Catalano Alessandro University of Padua, Padua, Italy

Hengerer Mark Ludwig-Maxmilians-Universität, Munich, Germany

Mayer Francoise Paul Valery University, Montpellier, France

Müller Leoš Stockholm University, Stockholm, Sweden

6.2 Languages and

literature

Achard-Bayle Guy University of Lorraine, Nancy, France

Balogh Andras Babeş-Bolyai-Universität, Cluj-Napoca, Romania

Raynaud Savina University Cattolica del Sacro Cuore, Milan, Italy

6.3 Philosophy, ethics

and religion

De Roover Jakob Ghent University, Ghent, Belgium

Müller Daniela University of Nijmegen, Nijmengen, Netherlands

Thomassen Einar University of Bergen, Bergen, Norway

10

Panellists

Panellists – country of residence

Country of residence Number Country of residence Number

Belgium 1 Norway 1

Czech Republic 6 Poland 1

Denmark 1 Austria 1

Finland 1 Romania 1

France 7 United Kingdom 4

Italy 2 Spain 1

Germany 4 Sweden 1

Netherlands 7 Ukraine 2

TOTAL 41

11

Average „success rate“ in contracting panel members: 124 addressed,

41 accepted, 33% success

Panels - recommendations

• Members of the panels are nominated by an institution responsible for evaluation

• Czech scientists will be full members of the subject panel, with their representation with a

maximum ratio of one Czech representative to two foreigners. Chairman of the panel is

always a foreigner, he/she will observe there are no conflicting issues stemming from the

presence of Czech panel members. The panel operates without advisers.

• The number of the panel members is related to the number of reviewed research units. This

is about 6 members for panels with up to 15 units, 9 members for 15 to 25 units, 12 members

when the number of units exceeds 25. The panel should not evaluate more than 40 research

units.

• In a remote assessment each unit is evaluated by three panellists (typically one Czech and

two foreigners), time for the assessment is from 0.5 to 1 day depending on the unit size.

• During the panel meeting the time specifically dedicated to the evaluation of one unit is

typically within 4 hours

• Number of panels can be expected between 30 and 35. For large scientific fields (e.g.,

physics or biology) two or three panels can be appointed, for small fields merging can be

introduced.

12

Self-assessment report („SAR“)

Submission Guidelines for the evaluated research organizations in pilot testing:

differences from the draft by Technopolis:

• No minimum number of participants (number of outputs / researchers)

• Introducing the organizational chart that specifies the context between EvU and RU

• Questions relating to the criterion I are on RU and EvU level or just on EvU level

• For a criterion of research excellence non-scholarly outputs were also accepted. There

is no strict condition for outputs in "English only„.

• A consistent form of HC tables for different types of RO

• Extending questions about post-gradual studies

• A brief description of the activity in the evaluated period is included

• A rationale for selection of EO-s required

13

Self-assessment report:

recommendations

• Structure: Narrative Section I, Data Section II

• Narrative: Summary - strategy, mission, goals (1 page); summary of past

activities over the evaluated period (3-4 pages), prospects for the next period (2-

3 pages)

• Data: blocks by 5 criteria, for the criterion I Environment RU will be presented in

the context of the EvU

• A list of all outputs linked with their authors will be presented, as well as a list of

researchers in the respective period (distinguishing those who are no longer

members during the evaluation or became members recently)

• Data to be normalized to FTE of researchers; need to establish clear rules to

calculate FTE researchers for Universities and for organizations whose

employees are engaged in science only partially

• The number of post-graduate students actually working in RU has to be clearly

stated14

Excellent outputs -

recommendations

• The number of outputs registered for evaluation will be specified,

outputs should not be restricted solely to the scientific field to which

the RU registered

• Non-scholarly excellent outputs should also be accepted, their

evaluation has to be specifically dealt with, descriptions in English have

to be provided to enable the evaluation by foreign reviewers

• If possible, outputs should be complemented with a bibliometric

analysis (WoS indicators, international review for SSH)

• Reviewers are selected by subject panels

• Two reviewers of the same level, while the third reviewer in case of a

difference of two or more points

• A new definition of "Grades", based on the definitions used in the pilot

testing

15

Bibliometrics -

recommendations

• Bibliometrics roughly as designed by TP, possibly can be simplified

• CZ Indicators: specify for the field where RU is registered and also

averaged by publishing a RU profile across disciplines

• Presentation of key indicators of bibliometric report also arranged in a

tabular form by RU ("matrix" form)

• Publications with authors from several RUs shall be considered for

each RU

• RIV expanded on book reviews

• Address the issue of taking into account the outcomes from applied

research (as done in pilot testing)

16

On-site visits

• They are not a part of the TP draft, mainly because of cost

• During SPE they were performed at all RUs; conclusion:

• The panels’ conclusive reflections were that the site visit was useful especially for

the institutions that under-presented their information in the self-assessment forms

or wrongly understood the information requirements. In the latter case, site visits

helped a lot in clarifying the aspects that RUs got wrong from the submission

guidelines. The panels suggested that in case site visits are absolutely out of the

question in the full-scale EM due to cost reasons, the evaluation agency may

consider organizing a Q&A session using online tools and/or videoconferencing.

(see Background report 10: The Small Pilot Evaluation – Feedback and Results, part 2.2.1)

• The prevailing opinion of the panellists during PT:

• visits are important or even indispensable

• „Hearing“: cheaper and in terms of organization easier option: visits of RU

representatives at the site of panel meeting

17

On-site vists - recommendations

• On-site visits – non-uniform positions within the IPN Metodika team as

well as among panellists and RUs

• the problem mainly in timing and logistics, and also in cost

• panellists mostly pro - personal contact between evaluators and

evaluated ones is desirable

• hearing as a compromise solution,

• require a firmly set agenda

18

Calibration

1st Interim Report (1IR):

• main panel chairs to ensure coherence in the interpretation of the assessment criteria and their key words

among the panels, so that, for example, a score 4 against the criterion ‘research performance’ has an

equal value in physics as in social sciences, for any type of organization. (1IR, part 4.8.1, page 80)

• Subject panels will undertake early calibration exercises to ensure that subject panel members (and

referees) develop a common understanding of the quality levels. The subject panels will continue to discuss

the application of the quality levels and will keep under review the scoring patterns of panel members

throughout the assessment process to ensure consistency in the subject panel’s standards of assessment.

(1IR, part 5.1.3, page 89)

PT:

• The only calibration meeting: the chairs and members of the main panel and the chairs / vice-chairs of

subject panels (May, Prague)

• Inadequate calibration is a weakness of PT: e.g. significantly different quality grades between different

subject panels

• Recommendations mentioned in the documents drawn up by Technopolis consortium should be expanded

and clarified

19

Calibration - recommendations

• Precisely define the number, the agenda and the participants of the

calibration meetings.

• In calibration meetings define the role and tasks of the main panel chair, and

the chairs of the subject panels and in implementing conclusions of these

meetings.

• Accurately describe the procedure that will guarantee the panels had a

uniform interpretation of the quality degrees (grades) in the evaluation by all

five criteria. Uniform interpretation must be ensured between the subject

panels within one disciplinary area as well as between different main

panels.

• Reassess the need for field-specific terms as suggested by Technopolis:

• significance, originality, rigour in criterion III "Scientific excellence"

• reach, significance in the criterion in the "Societal relevance"

• Reassess the need for the weight settings of sub-criteria in the criteria I, II,

and IV.

20

The evaluation report for EvU

• TP proposal is methodologically weak: „The Subject Panel Chair for the major field of

involvement of an EvU will be responsible for the analytical report at the EvU level, based upon a

draft developed by the Evaluation Management Team and in cooperation with the other relevant

Subject Panel Chairs“ (Summary report, part 3.2.4)

• TP template for the evaluation report for EvU has a form of a SWOT analysis

• A number of panelists expressed their doubts about the benefits of the

evaluation report for EvU; in addition, from self-assessment report it has not

been possible to obtain sufficient information for its creation

• At the panel meetings in Prague, it was agreed that EvU report will be

prepared by the chair of the main panel in collaboration with the chairs of

subject panels

21

EvU evaluation report -

recommendation

• It has to be clarified whether in a selected field-defined RU evaluation

methodology preparing of a report on the level EvU is possible and

necessary

• It is not clear how the report should look like and who will be in charge

of preparing it

22

Feedback from panellists

• Some comments already included in the above recommendations

• Similar regular evaluations of R&D are common in some European

countries (e.g. UK, FR, PT, IT, LT, LA)

• SSH:

• METODIKA introduces clear rules

• peer-review by foreign experts brings positive experience, limits potential bias of

national experts

• specific requirements for bibliometric report

• Definition of RU is somewhat questionable as RU goes across

organizational structure of EvU (exact sci.)

• There should be some way of checking on the truth content of what has

been written (in SAR)

• A clear idea of future vision and strategy to achieve it must be provided

23

Feedback from RUs/EvUs

• RUs consider present evaluation method (Metodika2013+) better,

simpler, more accurate, more objective (mostly exact sci., also some

humanities)

• RUs critical to the structure of the SAR (exact sci.), make it shorter

• 1-2 % excellent outputs is too little

• RUs critical to the bibliometric report (both exact sci. and humanities)

• Evaluation panel made a realistic assessment only after the site visit

• Out of 5 RUs in exact sci., 3 critical to site visit; in humanities mostly

positive

• Adequacy of the evaluation can be judged only when a certain minimum

number of RUs is taking part in

• Majority of RUs with panel´s conclusions/recommendations

24

Conclusions

• Evaluation based on methodology by Technopolis is feasible

• excerpts from feedback by panellists:

• performing a self-evaluation to be read by experts in itself can be a highly useful

exercise for an institution as well as for the individual researchers

• encourages institutions to assess their own performance against excellence defined on a

global scale. A result of this could be a better understanding by them of their competitive

positioning and could positively influence their goal setting

• Some modifications are needed, based also on experience from the on-

going evaluation by ASCR and on experience from METODIKA2013+

• Next evaluation should involve a smaller segment of RO-s rather than

all RO-s in the Czech Republic. Evaluation of all Czech universities is an

option

25

Thank you for your attention

ruzickav@tc.cz

www.metodika.reformy-msmt.cz