120.04.23

Making Good Use of Research Evaluations

Anneli Pauli, Vice President (Research)

220.04.23

Academy of Finland

Main tasks

320.04.23

Academy of Finland

Organisation

420.04.23

Academy of Finland

Evaluation activities

520.04.23

Benefits of using international peer review method

…to the Finnish Science

• Scientific quality guaranteed

• Easier to achieve objectivity, equal treatment and transparency

• Less negative feedback from researchers (small country)

• May enhance international collaboration

• More international visibility

620.04.23

Evaluation activities Impacts

• Applications

• Programmes

• Disciplines and

fields of research

• Research system

Excellent scientific quality is a prerequisite for scientific as well as societal relevance

Feedback received on the quality of research, scientific and societal relevance, interdisciplinarity, national and international cooperation, programme management;

Impact on the development of the field, identification of future research needs

Measures for further development of the field, prioritisation, possibly resource allocation

Knowledge on the structure and dynamics of science and innovation system for its further development

720.04.23

Peer review of applications

• Scientific rating by the peers

• Ranking by the Research Councils

• 35-100 % of peers are foreigners, depending on the call

• Applications in English

820.04.23

Research programmesFrom planning to implementation

• Follow-up and evaluation plans included in early stages of planning• A programme is evaluated against its objectives and funding volume• Main components of the evaluation

• added value of the programme, ‘programmatic nature’• scientific results and impact• societal, economic, technological, cultural impacts• implementation and management of the programme

• Utilisation of evaluation• after-care plans of the issues raised by the programme evaluation• identifying future research needs and directions

920.04.23

Research fields: impact of evaluations

• Thorough planning of the evaluation

• Timing of the evaluation

• Commitment of

• research community to develop the field (realistic expectations)

• other stakeholders (research councils, other funding organisations, ministries etc.)

• Follow-up

• within the process of management by results (Finnish Ministry of Education and Science and Academy of Finland)

• cooperation of researchers, research institutions, and other stakeholders

1020.04.23

CASE: Evaluation of Biotechnology in Finland (1)

• 2002: Evaluation of Biotechnology in Finland. Impact of Public Research Funding and Strategies for the Future.

• Commissioned by the Ministry of Education and Science • Coordinated by the Academy of Finland • Involved multiple stakeholders: ministries, Academy of Finland, National

Technology Agency of Finland Tekes, and Sitra, industry• Carried out by an international expert panel, chaired by Professor Fotis C.

Kafatos, then dir.gen. of EMBL

• 2003: A follow-up working group appointed by the Ministry of Education and Science (MoE)

• Made a proposal for future actions and implementation of the recommendations in 2003-2006

• 2005: MoE working group “Development programme for research and training in biotechnology in 2007-2009”

• Follows and evaluates the implementation of recommendations and measures based on the Biotech 2002 Evaluation and the work of the follow-up working group 2003

1120.04.23

CASE: Evaluation of Biotechnology in Finland (2)

2005, MoE working group: Questionnaire to universities and their biocentres, research institutes and funders of biotechnology:

• Information about resources and impact of research

• Implementation of the recommendations made in the BIOTECH 2002 evaluation• Which recommendations, how and when the organisation has

implemented?• Which recommendations and when will be implemented in the

future? • Which recommendations cannot be implemented and why? • Description of any other development measures which have been

carried out in 2002-2004. • Which of the recommendations or measures addressed to other

organisations have been of little importance and/or have been implemented inadequately?

1220.04.23

Research system

• Reviews on the State, Quality and Impact of Scientific Research (1997, 2000, 2003)

• Next 2005-2006:1. Bibliometrics

• Analyses of Finnish science (e.g. fields, organizations, cooperation)• Comparison of Finland to OECD countries’ performance

2. Impact evaluation• International benchmarking of evaluation methods and indicators• Academy’s four Research Councils: Impact assessments of funded

research External expert panel: Impact evaluation of the Academy

3. Foresight• Joint project of the Academy of Finland and the National Technology

Agency Tekes• 10 expert panels, important issues in the future• www.finnsight2015.fi

1320.04.23

Challenges: from planning to implementation

• What is the motivation for carrying out an evaluation

• Cost-effectiveness

• Selection of evaluators of crucial importance

• Stakeholders: multiple views, commitment needed

• Long-term planning vs. short-term needs

1420.04.23

Conclusions

• Different needs, levels, implementation and impacts

• Key issues of concern to research communities and institutions / policy makers / managers / others define the type of the evaluation

• Clear aims and criteria stated for each evaluation

• Right timing

• Results in a user-friendly format

• Combining evaluation with strategic planning / forward-looking / foresight

• Impacts can only be achieved if the findings of the evaluation results are utilised.