1 Programming period 2007-13 Strategy and Operational programmes DG REGIO – Unit B.3.
1 The best of both worlds: combining approaches Daniel Mouqué Evaluation Unit, DG REGIO.
-
Upload
amelia-warren -
Category
Documents
-
view
223 -
download
2
Transcript of 1 The best of both worlds: combining approaches Daniel Mouqué Evaluation Unit, DG REGIO.
1
The best of both worlds: combining
approaches
Daniel Mouqué
Evaluation Unit, DG REGIO
2
Cohesion Policy spending 2007-13
Other; €62bn Enterprise & innovation; €79bn
Transport; €76bn
Human capital; €68bn
Environment; €62bn
3
The magic bullet?
4
… More like a varied toolbox…
5
Tools of the trade include:
« Quantitative » (numbers are the narrative):– Counterfactuals/comparison groups
– Ex post CBA
– Micro or macro models
« Qualitative » (numbers feed the narrative):– Case studies
– Surveys
– Other « theory-based », eg rigorous observation
6
We need both approaches:
• Qualitative and quantitative
• Numbers and narrative
I will illustrate with examples…
7
Examples from evaluations
8
Example 1: enterprise support in Eastern Germany (2010)
9
WP6c of the ex post evaluation
• R&D & modernisation grants in E. Germany
• 2 databases: IAB enterprise « panel », GEFRA survey of innovation in Thuringia
• Not just before/after but comparison with matched non-assisted enterprises (PSM, etc)
• Mainly quant., but results discussed with focus groups including project and prog managers
10
Results: R&D grants
Impact of R&D grants
0
2000
4000
6000
8000
10000
12000
14000
Non-assisted firms Assisted firms
Inv
es
tme
nt
in e
uro
s/e
mp
loy
ee
Contribution from grant
Firm investment
11
Results: investment grants
Impact of investment grants
02000400060008000
100001200014000160001800020000
Non-assisted firms Assisted firmsInv
es
tme
nts
in e
uro
s/e
mp
loy
ee
Additional investmentfrom firm
Contribution from grant
Baseline
12
Results: employment• An estimated 27,000 jobs created
• Significantly lower than monitoring data for jobs created (107,000) and safeguarded (439,000)
before/after monitoring not a guide to impact main effect of grants is investment (and
productivity) change, not jobs This was not news to the focus group We would like more qualitative data to assess
why this happens/applicability elsewhere
13
Example 2: enterprise support in Italy (forthcoming)
14
• Grants to large firms, various support to SMEs
• Not just before/after but matched comparison enterprises (PSM, discontinuity design, etc)
• Early indication: SME support changes behaviour more. Eg €500k per job for large, €80k for SMEs
• In process of comparing different forms of support (grants, loans, VC, mentoring, combos)
• Beneficiary survey (1000 firms) shedding a lot of light on process
15
Example 3: Dutch innovation vouchers
(2007)
16
The vouchers
– credit note, worth max € 7500– Lottery: 100 winners, 944 losers (= control
group)– for SMEs only– no match funding required– application-oriented research questions,
placed with a defined group of institutes but no restrictions on level of question or technology
– valid for half a year
17
Results
• Projects from 87% of winners, only 8% of losers, so 79% additional projects
• But no significant repeats/persistence
For more qualitative methods:=> Why no persistence?=> Small effect on process innovation, no
sig. effects on product inno
18
Example 4: GDP growth at regional level
(Pellegrini, 2010)
19
-3
-2
-1
0
1
2
3
4
5
6
7
-3
-2
-1
0
1
2
3
4
5
6
7
Ann
ual %
cha
nge
in G
DP
per
head
, 199
5-20
06
A clear gap between the growth rates of Objective 1 and other regions, 1995-2006
GDP per head in PPS average 1988-1990 (EU15=100, log scale)
Source: "Measuring the Effects of European Regional P olicy on Economic Growth: a Regression Discontinuity Approach" Busillo, Muccigrosso, P ellegrini, Tarola, Terribile (2010)
NUTS II Objective 1 regions NUTS II non-Objective 1 regions
25 3 35 40 45 55 6 75 95 110 130 150 175 205 235
20
• Method shows clear impact
• But begs further questions:
– What difference between regions, regional strategies?
– Management efficiency and the delivery system?
– Soft impact (beyond GDP, local development)?
=> Need for sectoral evaluations, case studies etc
21
Example 5: ex post evaluation of URBAN 2
(2010)
22
URBAN 2?
• Ran from 2001 to 2006 (closed 2008)
• 70 programmes across EU14, 2.2 million people
• €754m ERDF, €1.6bn total
• Integrated approach to urban regeneration: physical, social, environmental
• Strong local development approach (local area, strong local partnership
23
Evaluation found lots of outputs
• 372 restoration projects (urban heritage)
• 2,314,000 m² of buildings converted and renovated (community centres, museums, libraries, creches)
• A further 557,115 m² developed for social, sports, education and health uses
• 3,238,000 m² of new green space
24
… and there’s more
• 5,980 SMEs/micro/new entrepreneurs supported (incubation, business services, microfinance)
• 108,000 people trained, > ½ from vulnerable groups (helped to overcome illiteracy, continue education, enter labour market for 1st time)
• 247 projects to reduce local crime, delivered in collaboration with community groups :
• street wardens
• CCTV, landscaping and street lighting
25
Case studies, interviews etc found:• Successful projects were:
– « Owned » and initiated by local partners
– Sustained by larger partners (and 60% of projects continue after URBAN II)
• In almost all cases, strong local perception of improvements due to URBAN 2
Excellent qualitative work and a great narrative, but…
26
Lack of good data blocked rigorous measurement of long term outcomesNotably in terms of:
• Employment/unemployment outcomes
• Enterprise performance
• Crime rates
=> No striking headline impact figures
(And wanted to do a counterfactual, failed for lack of beneficiary – not comparison – data)
27
In conclusion…
28
The quantitative techniques bring a lot to the table…
• Clear and convincing headline results
• Ability to quantify and « stratify »
• Harnessing the power of statistics
• Provide a « reality check »
29
… but so do more qualitative techniques
• The power of narrative
• Numbers should be the starting point for discussion, not the end of it
• “Not everything that counts can be measured, not everything that can be measured, counts”(Albert Einstein)