CONACYT

Seminar on evaluation of STI policies

Mexico City

June 19, 2014

S&T policy in Italy: The role of evaluation

Giorgio SirilliCERIS – CNR

ROARS

2

Outline of the presentation

Science policy in Italy and bejondThe most relevant evaluation exercises:

lessons learnedThe use of evaluation in S&T policy makingConcluding remarks

3

Italy

60 million inhabitants

4

Italy, a founding member of he European Union

Italy’s policies, including R&D policy, are heavily affected by the the European Union

5

Some data on Italy

R&D expenditure = 19,834 million euro

Researchers = 106,151 (full-time equivalent)

R&D/GDP = 1.25%

90 universities

40 Public Research Agencies (PRAs)

6

SectorR&D

expenditurePercentage

Government 2.723 13.8

Privare non profit 621 3.1

Business enperprises 10.813 54.5

University 5.677 28.6

Total 19.834 100,0

Source: ISTAT

R&D expenditure by performing sector - Italy 2012 (million euro)

The Lisbon target

66.6%

7

Sector Number Percentage

Government 18,780 17.7

Privare non profit 3,735 3.5

Business enperprises 39,808 37.5

University 43,828 41.3

Total 106,151 100.0

Source: ISTAT

Researchers by performing sector - Italy - 2011 (number in equivalent full time)

8

Science and technology policy in Italy: main actors

Junior minister for Research (1963)

Ministry of Universities and Research (1989)

Ministry of industry (Economic development)

Junior minister for Innovation (2000)

Ministry of Economy

9

Ministry of universities and research

Committee for Research Evaluation (CIVR)

National Committee for the Evaluation of the University System (CNUSV)

National Agency for the Evaluation of Universities and Research (ANVUR)

Universities

National Research Council

Other research agencies

Ministry of economic development

The evaluation system in Italy: main actors

Start: 2011

Members of the Editorial board: 14

Collaborators: 217

Contacts: 8 million (November 2011 – June 2014)

Average daily contacts: 8,000 November 2011; 13,000 in 2014)

Articles published: 1,627

Comments by readers: 24,880

ROARS is ranked 8° among the top cultural national blogs

ROARS, a genuine expression of democracy and participation, has become a very important player in the policy debate and in policy making

A new player in S&T policy

11

The neo-conservative wave of the 1980s

Ronald Reagan Margaret Thatcher

12

The new catchwords

New public management

Value for money

Accountability

Relevance

13

The UK

“All areas of public expenditure should demonstrate ‘value for money’”

Thatcher’s three Es:

economy

efficiency

effectiveness

14

The UK

Research Assessment Exercise (RAE)

Research Excellence Framework (REF) (impact)

“The REF will over time doubtless become more sophisticated and burdensome. In short we are creating a Frankenstein monster” (Ben Martin)

15

“Why shoud we payresearchers if we make the best shoes in the world?”Silvio Berlusconi

“Culture does not provide food”(Con la cultura non si mangia)Giulio Tremonti

The neo-conservative wave in Italy in the 1990s

16

The neo-conservative wave in Italy in the 1990s

Letizia Moratti

Italian minister of education and research

“You first show that you use efficiently and effectively the public money, then we will open the strings of the purse” - Never happened

17

Science policy in Italy

The neo-liberal approach after Reagan and ThatcherThe legitimation of R&D and higher education

18

The Entetreneurial State

The “visible hand” of the state

Innovationsmicro hard disksiliconmulti touchinteernetgpssirilcd screenlitium batteries

19

Science policy in Italy

The neo-liberal approach after Reagan and ThatcherThe legitimation of R&D and higher educationThe autonomy of University and PRAs

- the (mis)use of autonomy- the impact on the scientific community

The reduction of resourcesThe role of evaluation as a S&T policy instrument

20

Is the Italian scientific system efficient and effective?

The answer is YES

21

Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 6.1)

(dati ISI Web of Knowledge, Thomson-Reuters)http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf

# p

ap

ers

/million

US

D (

PP

P)

Number of publications (2004)/unit of R&D expenditure

22

Average annual growth of publications (2004-2010) (%)

Fonte: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 3.2)•(dati ISI Web of Knowledge, Thomson-Reuters)

http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf

Number of citations to the Web of Knowledge articles (2004-2010)

Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 5.3)(dati ISI Web of Knowledge, Thomson-Reuters)

•http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf

24

Source: VQR 2004-2010 – Rapporto Finale ANVUR, Giugno 2013 (Tab. 5.3)(dati ISI Web of Knowledge, Thomson-Reuters)

•http://www.anvur.org/rapporto/files/VQR2004-2010_RapportoFinale_parteterza_ConfrontiInternazionali.pdf

# c

ites/m

illion

US

D (

PP

P)

Number of citations (articles 2004)/unit of R&D expenditure

25

The reduction of public resources for R&D

Data on government budget appropriations for R&D in Italy show that from 2009 to 2012 there has been a 9.8% decrease in absolute values (from €9,778 million to €8,822 million) and a 12.7% decrease in real terms

In particular, research funds granted to universities by the ministry of education and research have declined from €366 to €96 million; the Prin shrunk from €170 million in 2010 and 2011 to €38 million in 2012, and may drop to zero in 2014, while the Ordinary fund for higher education, which provides block funding for universities, shrunk from €7.5 billion in 2009 to €6.6 billion in 2013

26

The difficult situation of the National Research Council

In the period 2005 – 2014 the overall financing from the ministry of education decreased in real terms of 5.3% – the overall budget of the agency is about 1 billion euro.

The financing has witnessed a dramatic structural change: the block grant which is managed autonomously by CNR has decreased by 20.4% while the financing linked to specific projects has rocketed by 1,675%.

In 2014 the block grant of €500 million will not even be enough to cover the obligatory expenses (salaries and fixed costs) of €620 million. This means that CNR has lost its autonomy and that, in order to survive, the agency will have to activate contracts linked to research and technical services both from public and private sources

27

Established in 2011

A government agency, not an authority

The relationship with MIUR (Ministry of education, universities and research)

ANVUR activities:

1. Evaluation of the Quality of Research (EQR)

2. National Scientific Qualification (NSQ)

3. Accreditation of universities (AVA)

Evaluation

28

Model: Research Assessment Exercise (RAE)Objective: Evaluation of Areas, Research structures and

Departments (not of researchers) Reference period: 2004-2010Start: 2011Actors:- ANVUR (National Agency for the Evaluation of Universities and

Research Institutes)- GEV (Evaluation Groups) (#14) (450 experts involved plus

referees)- Research structures (universities, research agencies)- Departments- Subjects evaluated: researchers (university teachers and PRA

researchers)

ANVUR Evaluation of the Quality of Research

29

Researchers’ products to be evaluated- journal articles- books and book chapters- patents- designs, exhibitions, software, manufactured items, prototypes, etc.

University teachers: 3 “products” over the period 2004-2010

Public Research Agencies researchers: 6 “products” over the period 2004-2010

Scores: from 1 (excellent) to -1 (missing)

ANVUR Evaluation of the Quality of Research

30

Outcome:- mobilisation of the scientific community- heavily criticised (methodologies, too fast)- a measure of effectiveness- considered a first exercise to be improved- used to assign resources by ministry

ANVUR Evaluation of the Quality of Research

31

Bibliometrics vs peer review

NSE, SSH

Journals: A, B, C

Median

Examiners

National Scientific Qualification

32

Was it worth?

69,000 applications

46,000 habilitated

1,100 legal actions

cost per habilitation (5,000 euro per candidate habilitated; 25, 000 euro per candidate hired)

the link between VQR and ASN

National Scientific Qualification

33

Under way

Strong opposition from universities

Accreditation of universities

34

Too powerful?

Too bureaucratic?

Playing a political game?

The role of ANVUR

35

The relationship between ANVUR and ROARS

A “frank”, and sometimes tough, dialogue

ROARS’s methodological suggestions often accepted

Some estimates

VQR 300 million euro (180,000 “products”)

NSQ: 1126 million euro (46,000 candidates)

RAE: 126 million euro (180.000 “products”)

ANVUR budget: 9 million euro

Rule of thumb: less than 1% of R&D budget devoted to its evaluation – this is not often the case

Evaluation is expensive

37

Knowledge: the bundle

University

teaching

research

“third mission”(60 indicators)

The learned society

Research agencies

research

problem solving

management

The Frankenstein monster (Martin)

League tables

Competition vs cooperation of scientists

Opportunistic behaviour

The split of the academic community (the good and the bad guys)

Peer review vs bibliometrics

NSE vs SSH

The threat of the equilibrium amongst the components of the bundle (research at the expense of other activities)

The other side of the coin of evaluation

Performance based research funding systems are national systems of ex-post university research output evaluation used to inform distribution of funding

Performance based research funding systems

41

“The rationale of performance funding is that funds should flow to institutions where performance is manifest: ‘performing’ institutions should receive more income than lesser performing institutions, which would provide performers with a competitive edge and would stimulate less performing institutions to perform. Output should be rewarded, not input.”

Herbst, 2007

Performance based research funding systems

The rationale

42

Share of university funding dependent on Performance-Based Research Funding Systems

Country Share (%) of what

Australia 6 Total revenue

Italy 2 Block grant

New Zeland 10 Block grant

Norway 2 Total funding

Slovak Republic 15 Total funding

UK 25 Research support

Hicks D., Reseach Policy (2012)

43

Share of university funding dependent on Performance-Based Research Funding Systems

Hicks D., Reseach Policy (2012)

“The distribution of university research funding is something of an illusion”

“It is the competition for prestige that creates powerful incentives within university systems”

“Performance-based research funding systems aim at excellence: they may compromise other important values such as equity and diversity”

“Control by professional elites”

44

Few evaluations carried out in the last years

Some analyses of innovation policies using innovation data (CIS)

Some analyses of the impact of a specific measure for the financing of R&S in firms through a counterfactual method

Additionality difficult to evaluate

Impact difficult to evaluate

No major pressure (or interest) from policy makers to evaluate support to industry

Evaluation of public policies and programs in Italy

45

Main methodological results of the Italian experience:

- impact difficult to measure

- very few exercises carried out

- indicators need improvement

Evaluation of the “Plans for development of scientific and technological networks”

46

Concluding remarks

The Italian S&T system is under pressureLegitimation of S&T and higher educationEvaluation as a key policy instrumentEvaluation was introduced lately but it is expected to stayEvaluation was used in an ideological wayEvaluation exercises heavily criticised from a

methodological point of viewImpact on the scientific community and on researchers’

behaviorToo much evaluation is harmfulMisuse of the results of evaluationEvaluation is expensiveConcentration in “excellent” institutions or spreading?Paradox: the S&T Italian system is good

Thank you for attention