Evaluation of Large Scale RDI...

Evaluation of Large Scale RDI

Infrastructures

Massimo Florio

Department of Economics, Management and Quantitative Methods

University of Milan

EIB INSTITUTE’S BROWN BAG SEMINARS

June 22, 2016

Background

June 2012

EIB Institute

Call for proposals

2012/C 162/10

EIBURS - EIB University

Research Sponsorship

Programme

Proposal by

University of Milan and CSIL

Cost-Benefit Analysis in

the Research, Development

and Innovation Sector

December 2012

December 2015

OBJECTIVE OF THE SEMINAR:

DISCUSSING THE

FINAL RESULTS OF THIS PROJECT

“(…)The current methodology within the Bank is

conservative and aligned to DG Regio’s ‘Guide to

cost/benefit analysis of investment projects’. However,

this document is itself deficient in its explanation of the

RDI sector.

(…) The aim of the project is to build on the current

internal review and produce a model which is

academically credible, robust but at the same time easy

to apply and easy to assess.”

History of CBA: France and USA

Jules Dupuit (1804 – 1866)

“De la mesure de l'utilité des travaux publics”

(Annales des Ponts et Chaussées, 1844)

History of CBA: France and USA

France, 1850's

• Ecole des Ponts et Chausseès

• Dupuit, Arsène Jules Étienne Juvénal, De la mesure de

l’utilité des travaux publics, Annales des Ponts et

Chaussées (1844)

USA, 1930's:

• Flood Control Act (1936)

USA, 1950's: attempts to codify the benefit-cost rules

• U.S. Army Corps of Engineers, Department of Agriculture,

Bureau of Reclamation, Federal Power Commission

• Proposed practices for economic analysis of river basin

projects - Green book (1950)

History of CBA:

international organisations and the UK

• Guidelines for project evaluation, UNIDO (1972)

• Project appraisal and planning for developing

countries by I.M.D. Little and J.A. Mirrlees, OECD

(1974)

• Economic analysis of projects by L. Squire and H.G.

van der Tak, World Bank (1975)

• UK, The Green Book – Appraisal and Evaluation in

Central Government (2003)

• OECD, Cost-Benefit Analysis and the Environment,

Pearce et al. (2006)

• EIB, The Economic Appraisal of Investment Projects

at the EIB (2013)

History of CBA: European Commission

1994: 1st edition, 28 pages

1997: 2nd edition, 84 pages

2008: 4th edition, 257 pages



A novelty of the fifth edition of

EC CBA Guide:

Ch. 7 - Research, development

and Innovation

8

Some information on CBA international practice are drawn from the results of a

survey conducted on selected OECD countries addressing the actual use, practice

and role of CBA in ex-ante project appraisal.

OECD, Government at glance

July 2015

http://www.oecd.org/gov/govataglance.htm

Rail (e.g. Austria, Denmark, Canada, Sweden,Netherlands).

Urban transport (e.g. New Zealand, Austria,Denmark, Canada, Sweden, Netherlands)

Airports, ports and waterways (e.g. Austria,Canada, Sweden, Netherlands, UK)

Water supply and wastewater (e.g. Canada, Netherlands)

Solid waste management (e.g. Canada, UK)

Other environmental projects: risk prevention and mitigation, natural asset conservation, etc. (e.g. Canada, Sweden, UK)

Energy: production, transmission anddistribution (e.g. Denmark, Canada, Sweden)

Education (e.g. Canada, UK)

Culture and leisure (e.g. NewZealand, Canada, UK)

ICT: telecommunications, broadband, ICTapplications to businesses and citizens(e.g. Canada, UK)

Health (e.g. Canada, Sweden)

Scientific research (e.g. Canada, UK)

Technological development andinnovation: science parks, technologicalparks, incubators, etc. (e.g. Canada, UK)

CBA for research infrastructures




The use of CBA in the practice: evidence from the OECD

Canada: All sectors,

including RDI

UK: All sectors,

including RDI

Sweden: Rail, Urban Transport, Airports, ports,

waterways, Energy, Health, Other environmental projects

The Netherlands: Rail, Urban Transport, Airports, ports,

waterways, Water supply

Denmark: Rail, Urban Transport, Energy

New Zealand: Urban Transport, Culture and

Leisure

Austria: Rail, Urban Transport, Airports,

ports, waterways

Key ingredients:

Long-run timeframe

Incremental approach

Shadow prices Risk assessement

Long-run timeframe Public infrastructure projects need long-term forecasts. For example,

o Railway: 30 years

o Energy: 15-25 years

o Broadband: 15-20 years

This makes long-term forecasts and intertemporal analysis crucial.

RDI infrastructure are not different.

Typical time horizon is often 25 years or

more.

Source: EC CBA Guide, 2014

Key ingredients

Key ingredients

Incremental approach

Markets do not price basic research: a public good not generating

profits.

applied research: imperfect markets because of asymmetric

information, externalities and risks.

applied research: imperfect markets because of asymmetric

information, externalities and risks.

Shadow prices are defined as the marginal social value of a good.

empirical proxies: marginal costs or willingness to pay, or a

combination of the two concepts.

Shadow prices

Key ingredients

Risk assessement

Key ingredients

Results of a Monte Carlo simulation

Examples of probability distribution and cumulated distribution of the ENPV

Values ENPV ERR

Base case 36.6 7.56%

Mean (expected value) 41.3 7.70%

Median 37.7 7.64%

Standard deviation 28.6 1.41%

Minimum value -25.9 3.65%

Central value 55.2 7.66%

Maximum value 136.3 11.66%

Probability of the ENPV being lower than zero or ERR being

lower than the reference discount rate

0.053 0.053

Probability distribution Cumulated distribution

The CBA model (1)

The expected economic net present value of the RDI infrastructure

[𝔼(𝐸𝑁𝑃𝑉𝑅𝐷𝐼 )] over the time horizon (T) is defined as the

difference between expected benefits and costs valued at shadow

prices and discounted at the social discount rate (r).

The model breaks down intertemporal benefits into two broad

classes – use and non-use benefits – and compares these benefits

with costs.

The expectation operator implies that all critical variables are

considered as stochastic.

All the benefits should be related to the main economic agents:

firms, consumers, employees, taxpayers.

The CBA model (2)

= Use benefits

= Non Use benefits

= Costs

)

The present value of COSTS is the sum of the:

• economic value of capital ( )

• labour cost of scientists ( )

• other administrative and technical staff ( )

• other operating costs ( )

• negative externalities if any ( ).

The CBA model: Costs and Benefits

5

The present value of

BENEFITS

is the sum of the:

• Firms ( )

• Employees ( )

• Users (A + S + C)

• Taxpayers (

(QOV + EXV)

5

It was built (1993-2008) by CERN.

It is located in a 27 km-long underground tunnel near Geneva.

In operation since 2009, it has discovered the Higgs Boson in 2013.

TIME HORIZON 33 years: 1993 - 2025

UNIT OF ANALYSIS the LHC and four detectors (collaborations)

SOCIAL DISCOUNT RATE 3% in real terms (adopted by the EC CBA Guide, 2014)

SHADOW PRICES Proxied by marginal WTP or marginal costs

COUNTERFACTUAL Business as usual scenario

QUASI-OPTION VALUE assumed 0

NEGATIVE EXTERNALITIES assumed 0

KEY PARAMETERS FOR THE CBA

Mont Blanc

Switzerland

France

The Large Hadron Collider (LHC)

Applications of the model

Consumers

Firms owners

Taxpayers

Employees

Benefits Customary partition of economic agents in the applied welfare economics literature:

Some evidence from literature: • Drèze, J. and Stern N. (1990). Policy reform, shadow prices and market prices, Journal of Public

Economics. • Johansson, P-O and Kriström, B. (2015). Cost-Benefit Analysis for Project Appraisal, Cambridge:

Cambridge University Press.

• Firms: profit maximization (producer surplus).

• Consumers: maximizing their utility (consumer surplus).

• Employees: maximizing their income for a given amount of efforts.

• Tax-payers: adjusting their decisions as a consequence of the existing fiscal constraints to minimize the burden of taxation.

The CBA model: Benefits

Technological Externalities

6

The value of presetn technological spillovers ( ) is given by:

• The discounted incremented social profits ( ) generated by companies

(j) of the RDI’s supply chain which have benefitted from a learning effect;

• And other externatilities.

tT

P jt

Benefits to suppliers Sample of 300 orders by purchase code compared with all LHC orders

LEGEND OF CERN ACTIVITY CODES 11 building work

12 roadworks

13 installation and supply of pipes

14 electrical installation work

15 heating and air-conditioning equipment (supply and installation)

16 hoisting gear

17 water supply and treatment

18 civil engineering and buildings

21 switch gear and switchboards

22 power transformers

23 power cables and conductors

24 control and communication cables

25 power supplies and converters

26 magnets

27 measurement and regulation

28 electrical engineering

29 electrical engineering components

31 active electronic components

32 passive electronic components

33 electronic measuring instruments

34 power supplies - transformers

35 functional modules & crates

36 rf and microwave components and equipment

37 circuit boards

38 electronics

39 electronic assembly and wiring work

41 computers and work-stations

42 storage systems

43 data-processing peripherals

44 interfaces (see also 35 series)

45 software

46 consumables items for data-processing

47 storage furniture (data-processing)

48 data communication

51 raw materials (supplies)

52 machine tools, workshop and quality control equipment

53 casting and moulding (manufacturing techniques)

54 forging (manufacturing techniques)

55 boiler metal work (manufacturing techniques)

56 sheet metal work (manufacturing techniques)

57 general machining work

58 precision machining work

59 specialised techniques

61 vacuum pumps

62 refrigeration equipment

63 gas-handling equipment

64 storage and transport of cryogens

65 measurement equipment (vacuum and low-temperature technology)

66 low-temperature materials

67 vacuum components & chambers

68 low-temperature components

69 vacuum and low-temperature technology

71 films and emulsions

72 scintillation counter components

73 wire chamber elements

74 special detector components

75 calorimeter elements

8a radiation protection

n.a. not available

STEP 1. IDENTIFICATION OF HIGH-TECH ORDERS

ACTIVITY CODES FOR HIGH-TECH ORDERS

POWER CABLES AND CONDUCTORS CASTING AND MOULDING (MANUFACTURING TECHNIQUES)

MAGNETS FORGING (MANUFACTURING TECHNIQUES)

MEASUREMENT AND REGULATION PRECISION MACHINING WORK

ELECTRICAL ENGINEERING VACUUM PUMPS

ELECTRICAL ENGINEERING COMPONENTS REFRIGERATION EQUIPMENT

ACTIVE ELECTRONIC COMPONENTS GAS-HANDLING EQUIPMENT

PASSIVE ELECTRONIC COMPONENTS STORAGE AND TRANSPORT OF CRYOGENS

ELECTRONIC MEASURING INSTRUMENTS MEASUREMENT EQUIPMENT (VACUUM AND LOW-TEMPERATURE TECHNOLOGY)

POWER SUPPLIERS - TRANSFORMERS LOW-TEMPERATURE MATERIALS

FUNCTIONAL MODULES & CRATES VACUUM COMPONENTS & CHAMBERS

RF AND MICROWAVE COMPONENTS AND EQUIPMENT LOW-TEMPERATURE COMPONENTS

CIRCUIT BOARDS VACUUM AND LOW-TEMPERATURE TECHNOLOGY

ELECTRONICS OPTICAL AND X-RAY EQUIPMENT

ELECTRONIC ASSEMBLY AND WIRING WORK


STEP 2. PROBABILITY DISTRIBUTION SHARE OF HIGH TECH PROCUREMENT- CERN

Empirical marginal Probability Density Function ; N = 10,0000

STEP 3. PROBABILITY DISTRIBUTION SHARE OF HIGH TECH PROCUREMENT- COLLABORATIONS


STEP 4. PROBABILITY DISTRIBUTION ECONOMIC UTILITY SALES RATIO


STEP 5. PROBABILITY DISTRIBUTION EBITDA MARGIN



TECHNOLOGICAL BENEFITS

ROOTTECHNOLOGICAL

TRANSFERGEANT4

•Multivariate analysis tool for very large datasets

•Available since1997

•External users: HEP community, industry

•Simulation software

•Available since1999

•External users: HEP community, spaceagencies, industry, hospitals

Licenses, start-ups, collaborationagreements

LHC

Benefits to software users


The present value of technological spillovers ( ) is given by:

• the discounted incremental social profits generated by companies ( ) of

the RI’s supply chain which have benefitted from a learning effect;

• and other externalities.


b) CERN external procurement -

commitment for total and high-tech orders a) Benefits to firms in the supply chain. Sample of 300 orders by purchase code

compared with all LHC orders

c) Distribution of EBITDA 2013 from ORBIS in firms at

NACE industry levels matched with CERN codes

d) ROOT download data

e) ENPV Cumulative distribution

function (Montecarlo simulation after

10,000 extractions) conditional to PDF

of critical variables (kEUR 2013)


12

• Objective: discovering whether there has been a long-term "learning

by doing" effect on CERN suppliers' profitability, beyond the initial

order.

• Access to CERN database on procurement data for LHC between

1995 and 2008. 1360 suppliers from 35 countries with least one order

of over CHF 10,000. A total of 11,969 orders. Out of the original list of

LHC suppliers, 1,060 companies have been identified in AMADEUS and

ORBIS. Companies (in total 500) - for which core financial indicators

over the time span 1991-2013 - were selected for a detailed analysis.

• Methodology entailed the collection of 23-year long time series of

financial data (1991-2013) for a large sample of companies, controlling

for time trends, firm-level and country-level possible confounding

factors.

Florio, M., Castelnovo, P., Sirtori, E., Rossi, L., Forte, S.

The economic impact of CERN procurement: evidence from

the Large Hadron Collider


13

• Research question: Is there any evidence in the data that the

profit margin of suppliers of the LHC increased over time

after the initial contract with CERN?

• Most interesting performance variable is the profit margin,

defined as the ratio of EBIT (Earnings Before Interest and

Taxes) to operating revenues. It is around 4.6% on average in the

sample.

• A dynamic panel data model was built. The following variables

were included amongst the controls: profit margins (lagged one

year), the firms’ total assets (control for size of the supplier) and

yearly GDP change in the country where the firm was located

and in the OECD area, a time trend variable, year fixed

effects.


Total CERN orders each year is the sum of the number of all the orders above CHF 10,000 in that year, including

multiple orders to individual firms. First time orders are the number of orders that were agreed with a firm for the

first time in that year. (First orders of 1995=2; 1996=17; 2008=4). Source: Processing of CERN data.

Total CERN orders for LHC (left scale) 5 44 443 646 818 866 1212 1068 2621 1504 1355 929 403 55

First time orders (right scale) 2 17 128 83 100 89 94 85 112 99 89 75 25 4


A: M

agn

ets

B: S

witc

h g

ea

r and

sw

itch

boa

rds

C: P

ow

er

cable

s a

nd c

ond

ucto

rs

D: C

ontr

ol an

d c

om

mun

ica

tion

cable

s

E: P

ow

er

supp

liers

an

dconve

rters

F:

Pa

ssi

ve e

lect

ronic

co

mpo

ne

nts

G:

Pow

er

sup

plie

rs -

tra

nsfo

rmers

H: R

F a

nd

mic

row

ave

com

pone

nts

I: R

aw

mate

rials

(su

pp

lies)

J: M

ach

ine t

oo

ls,

an

d q

ualit

ycontr

ol

K: C

asting

and m

ould

ing

L:

She

et m

eta

l w

ork

M:

Gen

era

l m

achin

ing w

ork

N: P

reci

sio

n m

achin

ing w

ork

O:

Spe

cia

lise

d te

chniq

ues

P: R

efr

igera

tion

equ

ipm

en

t

Q:

Sto

rage a

nd t

ranspo

rt o

fcry

og

ens

R: V

acuu

m a

nd

low

-tem

pera

ture

S: L

ow

-tem

pera

ture

mate

rials

T:

Va

cuum

com

pon

ents

&cham

bers

U: S

pe

cia

l dete

cto

rs c

om

pone

nts

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Percent share of number of orders. Minor items omitted. Source: CERN data

.


Distribution of LHC suppliers across countries

AT, 15

BE, 29

BG, 3 CA, 1

CZ, 8

DK, 9

FI, 14

FR, 401

DE, 148

GR, 5

HU, 3

IN, 3

IL, 3

IT, 101

JP, 5

LU, 3

NL, 33NO, 4

PL, 11

PT, 7RU, 15

SK, 4

ES, 37

SE, 15

CH, 371

UK, 77

US, 32

Others, 8

TOTAL (

SAMPLE

AT, 14

BE, 24

BG, 1CA, 1

CZ, 8 DK, 6

FI, 13FR, 272

DE, 138

GR, 5HU, 3

IN, 0

IL, 3

IT, 82

JP, 5LU, 3

NL, 33NO, 3

PL, 5

PT, 5RU, 8

SK, 3ES, 30SE, 14

CH, 279

UK, 71

US, 25

Others, 4

≅ 1000)

(≅ 500)


LHC suppliers distribution by NACE code

0 50 100 150 200 250

20. Manufacture of chemicals and chemical products

22. Manufacture of rubber and plastic products

23. Manufacture of other non-metallic mineral products

24. Manufacture of basic meta ls

25. Manufacture of fabr icated metal products, except machinery

26. Manufacture of computer, electronic and optical products

27. Manufacture of e lectrical equipment

28. Manufacture of machinery and equipment n.e.c.

30. Manufacture of o ther transpor t equipment

32. Other manufacturing

33. Repair and insta llation of machinery and equipment

35. Electricity, gas, steam and air conditioning supply

41. Construction of build ings

42. Civil engineering

43. Specialised construction activities

45. Wholesale and retail trade and repai r of motor vehicles

46. Wholesale trade, except of motor vehicles and motorcycles

47. Retail trade, except of motor vehicles and motorcycles

61. Telecommunications

62. Computer programming, consultancy

64. Financial service activities, except insurance

70. Activities of head offices; management consultancy

71. Architectural and engineer ing activities; technical testing

72. Scientific research and development

77. Rental and leasing activities

82. Office support and other business support activities

85. Education

Other (1)

n. of firms


average std. dev. 25th median 75th

Total sample 4.58% 12.94 1.21% 4.26% 8.45%

High-tech 4.62% 13.44 1.35% 4.46% 8.53%

Non-hi-tech 4.52% 11.98 0.95% 3.84% 8.15%

EBIT margin descriptive statistics

EBIT margin = EBIT/Operating Revenues. 25th and 75th are percentiles.

Source: Processing of CERN and AMADEUS-ORBIS data


Regression

results

EBIT MARGIN (%)

Full Sample High-Tech Firms Non High-Tech Firms

CERN Effect 0.673** 1.007*** 0.642

p-value (0.032) (0.001) (0.256)

Firm Level Controls

∆Total Assets1 YES YES YES**

∆EBIT margin, 1-year lagged

value YES*** YES*** YES***

Macroeconomic Controls

∆GDP growth rate, firm’s

country YES** YES* YES***

∆GDP growth rate, OECD Area YES YES YES**

Year fixed effect YES*** YES*** YES***

Time trend YES*** YES*** YES***

Constant YES*** YES YES***

Number of observations 4856 3017 1676

Number of Firms 333 203 118

Standard Errors are

clustered by country.

* p < 0.1, ** p < 0.05, *** p

< 0.01


Full Sample

0= Year of the 1st order

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

-15 -14 -13 -12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

EBIT margin mean

high tech

-2.5

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

-15 -14 -13 -12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

EBIT margin media

non-high-tech

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

-15 -14 -13 -12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

EBIT margin media


Focus on Technological impact

What can we learn from the LHC about the

impact on firms?

• Earlier studies at CERN were based on interviews

to managers. They concluded that for each Euro

of procurement there will be 3 Euro of sales to

other customers (or cost savings).

• Problems: not updated, selection bias, and

sales per se are not a social benefit.

• Lessons for the FCC Study.

Human Capital Formation

6

Human capital formation benefits (H) are valued as increased earnings 𝐈 gained by

RI’s students and former employees 𝐳 , since the moment 𝛗 they leave the

project, against counterfactual scenario.

Sector CERN

fellows

CERN

technical

students

CERN

doctoral

students

User-

students and

post-docs

Industry 20% 45% 20% 20%

Others

(computing,

finance, public

administration, …)

20% 45% 20% 20%

Research centres 30% 5% 30% 30%

Academia 30% 5% 30% 30%

TOTAL 100% 100% 100% 100%

Variable

Number over the 1993-2025 period

Average staying at CERN

CERN fellows working on LHC 5,873 2 years

CERN technical students working on LHC

3,940 1 year

CERN doctoral students working on LHC

1,332 3 years

User-students working on LHC 14,225 3 years

Post-doc researchers (users) working on LHC

11,301 2 years

TOTAL 36,671

Sources: - CERN personnel statistics; - Interviews to CERN staff

Main assumptions: - Future number of beneficiaries; - Number of users-students and post-docs among users (assumed based on their age group); - Incoming number of user-students and post docs

TYPES AND QUANTITIES OF PEOPLE

BENEFITTING FROM TRAINING

TYPES AND NUMBER OF PEOPLE

BENEFITTING FROM TRAINING

Post-docs (users 31-35 yrs old)

User-students (<30 yrs old)

Fellows

Technical students

Doctoral students

ESTIMATION OF FUTURE AVERAGE SALARIES DETERMINING THE RETURN TO SALARY DUE TO LHC TRAINING

SALARY EFFECT (1)

SALARY BONUS

FOR JOB EFFECT (2)

Sector CERN fellows,

doctoral students,

user students, post-

docs

CERN technical

students

Research centres

9.3% 2.5% Academia

Industry

Others (computing, financial, …)

(1) Survey to 192 former LHC students (out of a total survey to 385 students and former students): declared salary impact of the experience at LHC on their current salary

(2) Own assumption based on survey results and Payscale salaries

Main source: Findings from the survey to LHC current and former students

Main assumptions: • Same economic return regardless of the professional sector and type of student • Same return over the entire work career (40 yrs)

ASSUMED DISTRIBUTION OF FORMER LHC

STUDENTS BY PROFESSIONAL SECTOR

Industry: y = 12731ln(x) + 31792

Others: y = 14180ln(x) + 36165

Research, Academia: y = 9685.8ln(x) + 32575

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000

100,000

0 10 20 30 40

EU

R

Years of career

Industry

Others

Research centres

Academia

Log. (Industry)

Log. (Others)

Log. (Academia)


SHARE OF RESPONDENTS BY EXPERIMENT

SKILLS IMPROVED THANKS TO THE LHC

EXPERIENCE. AVERAGE JUDGEMENT

AN OVERVIEW OF CURRENT EMPLOYMENT

SECTOR. SHARE OF RESPONDENTS

AVERAGE SALARY EVOLUTION: A COMPARISON BETWEEN THE TWO

GROUPS OF RESPONDENTS (THOUSAND EUR)

37.9

64.7

84.7

38.3

75.8

94.4

0

10

20

30

40

50

60

70

80

90

100

Entry Salary Mid-Career Salary End-Career Salary

Respondentswho are currentlystudying orunemployed

Respondentswho are currentlyworking

THE IMPACT OF LHC EXPERIENCE ON SALARY (%)

ICT sector

(e.g. computing)

9%

ALICE5%

ATLAS22%

CMS65%

LHCb7%

Other1% 4.29

4.17

3.93

4.053.38

3.42

3.81

0.0

1.0

2.0

3.0

4.0

5.0Scientific skills

Technical skills

Communication skills

Problem-solving capacity

Team/project leadership

Developing, maintaining and

using networks of collaborations

Independent thinking/critical

analysis/creativity


Knowledge output

6

• The sum of the present value of papers signed by RDI’s scientists (P0t) and the value of subsequentflows of papers produced by other scientists that se or elaborate the RDI scientist’s results.

• Divided by the number of reference they contain (Pit / Kit wih i=1,…n) and the value of citation each paper receives, as a proxy of the social recognition that the scientific community ackonwledges to the paper (Qit with i=0,…n) .

The social value of knowledge output is measured by:

Knowledge output

6

41

The social value of knowledge output is measured by:

the sum of the present value of papers signed by RI’s scientists and the value of

subsequent flows of papers produced by other scientists that use or elaborate of the RI’s

scientists’ results

divided by the number of references they contain and the value of

citations each paper receives, as a proxy of the social recognition that the scientific

community acknowledges to the paper

Knowledge output

6

42

PAPERS PRODUCED BY LHC USERS (L0) PAPERS PRODUCED BY NON-LHC USERS (L1 & L2)

DOWNLOADS OF LHC PAPERS (D1)

0

500

1,000

1,500

2,000

2,500

19

93

19

95

19

97

19

99

20

01

20

03

20

05

20

07

20

09

20

11

20

13

20

15

20

17

20

19

20

21

20

23

20

25

Number of papers L0

L0, 2013-2025

L0, 1993-2012

Forecast

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

180,000

199

3

199

5

199

7

199

9

200

1

200

3

200

5

200

7

200

9

201

1

201

3

201

5

201

7

201

9

202

1

202

3

202

5

202

7

202

9

203

1

203

3

203

5

203

7

203

9

204

1

204

3

204

5

204

7

204

9

Number of papers L0, L1 and L2

L0, 1993-2025 L1, 1993-2050 L2, 1993-2050

Forecast

0

10

20

30

40

50

60

70

80

90

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

2012

2013

Number of downloads per paper (ArXiv, field HEP)

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

180,0001

99

3

19

96

19

99

20

02

20

05

20

08

20

11

20

14

20

17

20

20

20

23

20

26

20

29

20

32

20

35

20

38

20

41

20

44

20

47

20

50

Number of papers L0, L1 and L2 and downloads D1

L0, 1993-2025 L1, 1993-2050

L2, 1993-2050 D1, 1993-2050

Forecast

OUR RESULTS

VALUATION

TRACKING THE KNOWLEDGE OUTPUTS

Quantification of citations L1 Quantification of citations L2

Future number of citations

L2 per paper L0 = 4

Source: Preliminary scientometric analysis of INSPIRE database of papers and citations

42

Present value of papers L1

Present value of citations L1

Present value of citations L2

Present value of downloads

Unit economic value of papers L1 Value Source

Number of references in

paper L1 35

Own assumption, based on an analysis of 41

research journals by Abt and Garfield (2002)

Share of time dedicated

to research 65%

Own assumption. The remainder is for

teaching and other non scientific activities

Number of paper

(published

and non) per year

3.5

Own assumption. It represents the number of

papers to wich a scientist gives a real

contribution

Average annual gross

salary 59,289 €

Own elaboration based on PayScale data. It is

the average salary for a scientists working in

research centres and academia in the USA

Unit production cost per

paper L1 315 € = (59,289 € *

65%/3.5/35)

Own estimation, based on the approach

suggested by Florio and Sirtori (2014)

Unit economic value of citations and downloads Value Source

Working hours per year 1,800 = 225 working days *

8 hours/day Own assumption

Average hourly gross salary 33 € = 59,289/1,800 Own estimation

Hours per citation 3 Own assumption

Hours per download 3 Own assumption

Value of one citation L1 and

L2 99 € = 33 € * 3

Own estimation, based on Florio

and Sirtori (2014)

Value of one L0 paper

downloaded but non cited 99 € = 33 € * 3 Own estimation, based on Florio

and Sirtori (2014)

Except

Cultural Effects

Outreach activities carried out by RI produce cultural effects on the general

public 𝑔 , which can be valued by estimating the willingness to pay of the

general public for such activities.

Cultural Effects

Outreach activities carried out by RI produce cultural effects on the

general public 𝑔 , which can be valued by estimating the willingness to

pay of the general public for such activities.

Cultural Effects TRAVEL ZONES CONSIDERED

VALUATION THROUGH THE

TRAVEL COST METHOD

Origin

zone

Radius distance

from CERN

Share of

visitors

Source/

Assumption

Zone 1 500 km 24% CERN

Zone 2 500-1,500 km 50% Own assumption

Zone 3 Beyond 1,500 km 26% Own assumption

Main assumption: • % of visitors by mode of transport • Travel cost by zone

Source: HEATCO values of travel time by modes of transport

45

Zone 1

Zone 2

Zone 3

BENEFIT FOR SOCIAL MEDIA USERS

Main source: CERN staff

Main assumption: Future number of visitors

BENEFITS TO PERSONAL VISITORS:

QUANTIFICATION OF VISITORS

BENEFIT FOR WEBSITE VISITORS

Main assumption: Benefit = value of time spent on social media: approximate 2 minutes/hit

Main assumption: Benefit = value of time spent on social media

MASS MEDIA BENEFITS: NEWS BY MEDIA CHART

BENEFIT FOR VOLUNTEER COMPUTING

0

2,000

4,000

6,000

8,000

10,000

12,000

20

07

20

08

20

09

20

10

20

11

20

12

20

13

20

14

20

15

20

16

20

17

20

18

20

19

20

20

20

21

20

22

20

23

20

24

20

25

Number of volunteers - Six track

Number of volunteers - Test4 Theory

Main assumption: Benefit = Value of time spent to download, on forum.

SHARE OF BENEFITS BY TYPE OF OUTREACH ACTIVITY

The Non-use Benefit

where

• QOV is intrinsically uncertain and therefore not measurable, simply

assumed to be non-negative and then skipped;

• EXV, on the other hand, can be proxied by stated or revealed willingness to

pay for scientific research, and/or through benefit transfer, borrowing ideas

from CBA of the environment.

Bn captures two types of benefit related to the social value of discovery: the quasi-option value (QOV) and the existence value (EXV)

The Non-use Benefit 47

captures two types of benefits related to the social value of discovery:

the quasi-option value (QOV) and the existence value (EXV):

where

• QOV is intrinsically uncertain and therefore not measurable, simply assumed to be non-

negative and then skipped;

• EXV, on the other hand, can be proxied by stated or revealed willingness to pay for scientific

research, and/or through benefit transfer, borrowing ideas from CBA of the environment.

The Non-use Benefit 48

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ES FR IT UK

Humanistic Mixed Scientific n.a.

RESPONDENTS BY UNIVERSITY DEGREE RESPONDENTS BY LEVEL OF HOUSEHOLD INCOME GENDER Number

Female 581

Male 446

Total 1027

COUNTRY Number Italy 422

Spain 204

France 201

UK 200

Total 1027

YEARS Number 19-25 years 875

26-30 years 95

31-35 years 34

Over 35 years 20

n.a. 3

Total 1027

0%

20%

40%

60%

80%

100%

ES FR IT UK

up to EUR 1000 EUR 1000-3000 EUR 3000-5000

higher than EUR 5000 n.a.

RATING THE IMPORTANCE TO FINANCE RDI

Essential

38%

Important

46%

Rather

Important

14%

It is

insignificant

1%

It is not

necessary

1%

WHAT IS THE UTILITY OF THE LHC

0%

20%

40%

60%

80%

100%

ES FR IT UK

NO I DON'T KNOW YES

WILLINGNESS TO PAY FOR LHC

27%

8%

22%

43%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

0 0.5 1 2

EUR per person per year

AVERAGE ANNUAL WTP WTP TO PAY UNA TANTUM

0%

5%

10%

15%

20%

25%

30%

Average = 18%

SHARE OF ADULT POPULATION (18-74 YEARS OLD) WITH AT

LEAST TERTIARY EDUCATION

NO

49.1%

I DON'T

KNOW

36.6%

YES

14.3%

0

200

400

600

800

1000

It is a worthless infrastructurewhose construction could

have been avoided

It is an infrastructure ofinterest for physicists

It is an infrastructuredangerous because of the risk

of nuclear accidents

It is an infrastructure useful forthe production of the energy

It is a useful infrastructure toexperience accelerations

between protons that can beused for many purposes

LHC: CBA in a nutshell (1)

●

TOTAL MEASURED BENEFITS

LHC: summary of costs and benefits (Billion, EUR)

COSTS: 13.5 ± 0.4

USE BENEFITS:

Knowledge Formation 0.3 ± 0.1

Human Capital 5.5 ± 0.3

Technological Spillovers 5.3 ±1.7

Cultural 2.1 ± 0.5

NON-USE BENEFITS:

Existence Value 3.2 ± 1.0

Human capital, technological spillovers,

cultural + existence value each give

about 33% of benefits (publications are

negligible)

Uncertainty largest on technological

spillovers

More than 90% chance of positive NPV

Scientific publications 2%

Human capital formation 33%

Technological spillovers 32%

Cultural effects 13%

Existence value 20%

LHC: CBA in a nutshell (2)

●

ESTIMATED PARAMETERS OF

DISTRIBUTION

Mean 2,855,528

Median 2,825,860

Standard

Deviation

2,134,763

Minimum -6,220,259

Maximum 11,573,387

Estimated probabilities

Pr. ENPV ≤ 0 0.086

Montecarlo error

3 ϭ 10,000 0.02

PROBABILITY DENSITY FUNCTION

Provision of Services Social benefits of RDI services for target groups of consumers

Some Examples:

• NASA Glenn Research Center

• European Synchrotron Radiation Facility

• High Magnetic Field Laboratory in The

Netherlands

• Laserlab-Europe

Some RDI infrastructures provide services to external users. They may pay a fee for accessing and using the infrastructure’s equipment and/or specific services offered.

Some RDI infrastructures are expected to use new knowledge to deliver innovative services and products addressing specific societal needs. Benefits arise to users who are better off by the delivery of the innovative service or product.

Some Examples:

• Reduction of GHG and air pollutant emissions

• Improved energy efficiency

• Reduction in vulnerability and exposure to

natural hazards

• Improved health conditions

Social benefits to consumers of services


CNAO - National Hadrontherapy Center for Cancer Treatment

TIME HORIZON 30 years: 2001 - 2031

UNIT OF ANALYSIS

The hall hosting the particle accelerators and the other areas functional to the proper functioning of the clinical facility

SOCIAL DISCOUNT RATE 3% in real terms (adopted by the EC CBA Guide, 2014)

SHADOW PRICES Proxied by marginal WTP or marginal costs

COUNTERFACTUAL Do-nothing

Non-use benefits assumed 0

NEGATIVE EXTERNALITIES assumed 0

KEY PARAMETERS FOR THE CBA

It is an applied research facility specialised in hadrontherapy, an advanced oncological

treatment showing (in some cases) clinical advantages as compared to traditional radiotherapy.

It comprises two broad distinct areas: the high technology components, made of a set of

accelerators and transport lines of particle beams, and a clinical ‘day hospital’ facility,

comprising reception desks, waiting and changing rooms.

ENPV

Past investment costs

The ingredients of the CBA of CNAO

Past operating costs

Future operating costs

Future investment costs

Knowledge output

Technological spillovers

Human capital formation

Cultural effects

Health benefits

Decommissioning costs Provision of Services


# of protocol Clinical alternative Marginal percentage of patients who fully recover compared to the

counterfactual situation

1 No alternative 73%



9 Surgery + photon therapy 45%

10 Surgey 21%

11 No alterative* 45%


15 Surgery + photon therapy 30%

16 Photon therapy 43%


19 Photontherapy 36%

Type 1 – FULL RECOVERY Marginal benefit by protocols

# of

protocol Clinical alternative

Marginal percentage of patients

who fully recover compared to the

counterfactual

Number of life years

gained with respect to

the counterfactual

6 No alternative for advanced tumours 15% 5

8 No alterative 43% 3

14 No alterative* 68% 0.5

18 Palliative chemotherapy 40% 2

20 No alterative 43% 3

22 Surgey + photon therapy 10% 5

23 Photontherapy* 35% 5

Type 2 – INCREASE IN LIFE EXPECTANCY Marginal benefit by protocols

Type 3 – BETTER QUALITY OF LIFE Marginal benefit by protocols

# of

protocol Clinical alternative

Marginal percentage of

patients who fully recover

compared to the

counterfactual

Number of life years gained

with respect to the

counterfactual

Quality of life

adjustment factor*

7 No alterative 100% 1 0.3

21 Surgey 100% 15 0.3

Probability distribution of applied research benefits on patients (Euros)

Estimated parameters of the distribution

Mean 2,028,626,666

Median 1,984,699,763

Standard deviation 495,675,860

Minimum 935,508,430

Maximum 4,061,318,078


Pr. EPV ≤ base value 0.480

Pr. EPV ≤ 0 0.000

CNAO: Estimation of health benefits Applications of the model

CNAO: CBA Results PROBABILITY DISTRIBUTION OF

THE CNAO NET PRESENT VALUE

Own estimate of the Present Value PDF resulting from a Monte Carlo simulation (10,000 random extractions)

ESTIMATED PARAMETERS OF DISTRIBUTION

Mean 1,658,358

Median 1,615,046

Standard Deviation 499,225

Minimum 498,433

Maximum 3,686,989


Pr. ENPV ≤ 0 0.000

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

ENPV Probability Density Function

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

498,433,509 1,136,144,790 1,773,856,072 2,411,567,353 3,049,278,634 3,686,989,916

ENPV

ENPV Cumulative Distribution Function

Cumulated probability CBA reference value

Mean Median

Std. Dev. from mean

Carbon Ion Therapy

Proton Therapy

Revenues

Benefit of Technological Spillovers

Benefit of Human Capital Generation

Benefit of Knowledge Creation

Benefit of Cultural Outreach

74.2

20.9%

2.2%

1.1%

0.7%

0.6%

0.3%

Health

benefits

55/0

Values in Thousands EUR, 2013


Conclusions: Future Research

8

Testing the model on other Research Infrastructures

Forecasting technological spillovers with a control group

of firms (non-CERN suppliers)

Estimating human capital effects with econometric

‘treatment’ techniques

Developing a forecasting model for media impact of

outreach

Expanding the contingent valuation of willingness to pay

for discoveries

Applying the model to the Future Circular Collider

THANK YOU

[email protected] 27

Florio, M., Forte, S. and Sirtori, E.

2016

Forecasting the Social Impact

of the Large Hadron Collider:

A Cost-Benefit Analysis to

2025 and Beyond.

Available at

http://arxiv.org/abs/1603.00886

Forthcoming on Technological

Forecasting & Social Change,

Special Issue on the Social

Impact of Research

Infrastructures at the Frontier of Science and Technology

http://arxiv.org/abs/1603.00886

Evaluation of Large Scale RDI...

Documents

Transcript of Evaluation of Large Scale RDI...