1 Monitoring and Evaluation 2014-2020 – ERDF & CF A Stronger Focus on Results Veronica Gaffey DG...
-
Upload
rafael-frickey -
Category
Documents
-
view
215 -
download
1
Transcript of 1 Monitoring and Evaluation 2014-2020 – ERDF & CF A Stronger Focus on Results Veronica Gaffey DG...
1
Monitoring and Evaluation 2014-2020 – ERDF & CF
A Stronger Focus on Results
Veronica GaffeyDG REGIO Evaluation Unit
2
What is the problem?• Current programmes are often just designed to
spend.
– Objectives vague,
– How to recognize success or failure often not clear,
– Difference monitoring – evaluation not clear. Shortcomings in knowledge of evaluation methods
• Consequently, it is difficult to demonstrate value of the policy.
3
What is proposed?• More Concentration
• A focus on results (not only spending)
• Programmes with clear articulation of what they aim to change and how
• Better gathering of basic data on outputs
• Performance Framework and Reserve to incentivise performance
• Evaluation – more focused ex ante; evaluation plan obligatory, evaluation of the effects of each priority during the programming period; summary of evidence in 2020; ex post by Commission
• Draft Guidance: Concepts and Recommendations at http://ec.europa.eu/regional_policy/impact/evaluation/index_en.cfm
4
Result orientation – what is a result?For each priority axis
• Each priority axis shall articulate what it wants to change. What motivates the policy?
– E.g., accessibility of a region, productivity of SMEs
• Express the dimension of change with a result indicator.
– Reduction in travelling time; productivity of SMEs in region or productivity of supported SMEs compared to regional benchmark
• Define baseline: situation before programme start.
• Target: Where does the region or MS want to be at the end of the period in relation to the result indicator? (quantitative or qualitative e.g., desired direction of change, range of values, objective description of situation –removal of obstacle)
5
II Result indicators: ex ante conditionality
• Annex IV of CSF regulation – task for MS
– Existence of statistical system to undertake evaluations to assess effectiveness & impact of programmes
– Existence of an effective system of result indicators necessary to monitor progress towards results & to undertake impact evaluation
• Quality criteria: robustness, clarity of normative interpretation, responsiveness, timely collection, public availability.
• Identification of sources, establishment of targets
6
Intervention logic and outputs What factors are likely to affect the result
indicator? Road condition, traffic management system,
…
Select the factor(s) that the programme should influence – policy actions – what outputs will they produce?
E.g., improved roads – measured in km No baselines (“zero”) Set the target
7
Requirements for Indicators• Programme Specific Indicators:
– Outputs – baselines zero, cumulative values, quantified targets
– Results – baseline and quantitative or qualitative target
• Common Indicators (mostly output):
– Baselines zero,
– Cumulative values,
– Quantified Targets
8
Common Indicators
• Objective: provide EU level (aggregated) information on achievements of Cohesion Policy
• Timing: information should be available at or shortly after finishing projects (selected and fully implemented)
• Simplicity: data from monitoring of implementation or operation
• Aggregation: “outputs” as much as possible
9
Common Indicators in Regulations
• General Regulation: obligation to use common indicators
• Fund-specific regulations: specify relevant areas, name and measurement unit
• Guidance documents: – provide broad definitions– other information (type: result/output)
10
Performance framework I• Aims to support the progressive achievement of programme
objectives
• Sets out milestones (interim steps) and targets for performance of programme priorities for 2016, 2018 and 2022 – in each OP & consolidated in Partnership Contract
• Milestones for 2016 to include financial indicators and output indicators
• Milestones for 2018 to include financial indicators, output indicators and where appropriate, result indicators (under the control of Managing Authorities – possibly common indicators)
• Milestones also for key implementation steps
11
Performance Framework II• Milestones should be:
– Relevant (to the objectives of the priority)
– Transparent (objectively verifiable targets and data sources identified and publicly available)
– Verifiable, without disproportionate administrative burden
– Consistent across OPs where appropriate
• Ex ante evaluation appraises all indicators (logic of intervention and appropriateness of targets) and the suitability of milestones for performance framework
12
Performance review
• Performance reviews in 2017 and 2019 and the disbursement of reserve in 2019 based on Performance Framework
• Information for the performance review in the AIRs and progress reports of the Partnership Contract
• Member States are expected to react to significant shortfalls in the achievement of milestones (measures to improve performance, reprogramming)
• In the absence of sufficient action, Commission can suspend payments
• Significant failure (to be defined in implementing acts) to achieve the targets set for 2022 in the performance framework can lead to a financial correction at the end of the programming period
13
Performance reserve
• 5% of resources set aside at the beginning of the programming period (exception for ETC)
• The performance reserve is established per CSF Fund, per Member State and per category of region – NB – no competition between MS
• The 5% reserve is allocated to each Member State following the performance review in 2019
• Allocation can only be used for priority axes where performance has been satisfactory (milestones have been achieved) – based on Member State proposal
14
Priority axisIndicator and measurement
unit, where appropriateMilestones
for 2016Milestones
for 2018Target for
2022
Enhancing R&I infrastructure and
capacities to develop R&I excellence and promoting
centres of competence
Expenditure (EUR million) 60 180 420
Size of developed infrastructure completed (m2)
100 1000 1500
Researchers employed in enhanced infrastructure
100
Promoting entrepreneurship, in
particular by facilitating the economic exploitation of new ideas and fostering the creation of new firms
Expenditure (EUR million) 3 9 22
Number of New Enterprises supported
60 180 450
Gross jobs created in assisted SMEs
540 1400
Developing comprehensive high
quality and interoperable railway system
Expenditure (EUR million) 5 100 250
Feasibility studies completed and contracts for construction
concluded
Yes
Length of rail completed (km) 0 20 50
15
What should be in AIR? Result indicators: Progress towards targets Output indicators:
• Selected projects,• Fully implemented projects
• Actions taken to fulfil ex ante conditionalities• 2017 & 2019: performance framework and
progress towards achieving objectives of the programme
• 2019 and FIR: contribution to EU2020 objectives
16
Evaluation: when?
Ex ante evaluation: MS During the period: MS
Based on obligatory evaluation plan Adopted by monitoring committee
Ex post: Commission
17
Evaluation: what?
Impact evaluations – at least once during the period for each priority axis Does the intervention work? Why and how does the intervention work (or
not)?
Implementation evaluations (roughly what has been done so far)
Guidance on methods: EVALSED
18
Next steps
Concepts and recommendations discussed with MS, later adaptation to agreed regulations.
MS have asked for guidance on ex ante evaluation and performance framework
Work with MS / geographic units in 2012 on result indicators.
19
Example of result indicators – Annex 2 of
Concepts and Recommendations Paper
20
Result indicators Example: support to
enterprise
21
Region x has a problem…
SME productivity(source: national statistics)
0
20
40
60
80
100
Region X National average
22
… and a solution
• Grants to SMEs(capital equipment, modernisation, etc)
• Let’s assume the project runs well, the money is absorbed, there are outputs (no. of firms, etc)Is this enough? Do we know it has worked?
23
How do we know it works?
Need a result indicator. Let’s try 2 classics:
1. Jobs created in assisted projects/firms?But jobs created not the main effect of grants
2. Change in regional GDP/head?But long (and possibly unclear) link to measure
More fundamentally:Neither of these was the policy target
24
Regional SME productivity(defined as GVA/worker)
• This an appropriate result indicator
Not rocket science, but how common in practice?
• Set a target: 80% => 85% of national average
• Source: regional statistics (note time lag)
Grant,monitoring
system
Productivityof SMEs
supported
RegionalSME
productivity
25
In sum
We’re asking regions to develop a clear causal chain:
• Identify a problem
• Implement a possible solution
• Monitor an appropriate result indicator
• Seems obvious, but new in practice
• Still need evaluation to assess policy contribution to the result (ie “impact”)
26
Result indicators
Example: road network construction
27
Better accessibility wanted
• The baseline for the infrastructure programme as a whole is the value X of the road accessibility index – known from a existing transport model
• The MS aims to reduce the index value for the 3 most lagging regions by about 15% within the programming period (the target for the result indicator).
• Impact on what? - Road accessibility index
28
… and how to get it
• Expansion of highway network, including the construction of a last missing project in the trans-European network.
• 2 out of 3 TEN-T highway projects are completed, 70% of the envisaged road length.
• Envisaged output: 200 km of new highway + access roads
29
Monitoring and evaluation
Monitoring of outputs- Km in selected projects + actually built
Monitoring of result indicator- The accessibility index will be modelled in years 4 and 8.
Evaluation. The evaluation plan comprises: – Use of sectoral model. The models allows to isolate the effect
of key projects financed under the programme on the accessibility index.
– ex post cost benefit analyses for key projects co-financed in the previous programming period;
Data required Road transport data, collected by regular national and regional surveys