2_Monitoring and Evaluation of CAP 2014 – 2020

Approach of PII1

Jela Tvrdonova 2015

Contents Why do we need a common approach to M&E of

common agriculture policy? Who is responsible? How does the policy and the legal framework look like? How is the M&E system built up and what guidance is

provided? Evaluation questions Indicators Evaluation approach What lessons can be drawn from the assessment of

impacts in rural development of the current programming period?

2

When?

Who is responsible?

4/5 June 2009

Challenges of the CMEF & Ongoing Evaluation

3

One policy = one evaluation→ CAP common impacts

Pillar I

European Commission

Pillar II

Member States

Need for coordination and collaboration

Who?How?

Monitoring & Evaluation

For 2007-2013 the CMEF (Common Monitoring and Evaluation Framework) relates only to rural development.

For 2014-2020 there will be one monitoring and evaluation framemork for the whole CAP (Art. 110 of CAP Horizontal Regulation proposal).

Important components will be (among others) indicators and common EQ

4

Legal framework for the evaluation

Objectives of monitoring and evaluation of the CAP

5

RDR, Art.75: Monitoring and evaluation system aims at:

Improving the quality of the design and the implementation programmes;

Demonstrating the progress and achievements of rural development policy;

Assessing the impact, effectiveness, efficiency and relevance of the RD policy interventions.

Technical Handbook of the M&E framework of the CAP

Provides information on:

Objectives and purpose of M&E,

Framework for M&E of the CAP (intervention logic and indicators),

Actors and responsibilities in M&E (incl. Expert group of M&E the CAP),

Data sources,

Rural development specificities for monitoring (MC, AIR, PF) and evaluation (EP, ex ante, CEQ, ex post), including links to guidance documents already published,

Use of M&E information,

Future development,

Annexes: pillar I and II indicators fiches.

6

CMES as part of CMEF

7

CAP Common Monitoring and Evaluation Framework (1306/2013, Art. 110)

Common elementsPillar I specific elements Pillar II specific elements

Responsibility for evaluation: DG Agri

Common Monitoring and Evaluation System (1305/2013, Art. 67 and 808/2014. Art.14)

Common impact indicators

Data sources

Common contextCommon context indicators

Intervention logic: Overall CAP policy objectivesCommon impacts

Pillar I specific objectivesPillar I instruments

Pillar I result indicators Pillar I output indicators

RD priorities/specific objectivesPillar II measures

Pillar II result /target indicators Pillar II output indicators Performance framework

Operation database

Responsibility for evaluation: MAs

Common evaluation questions

Elements of the CMES (Reg.808/2014, Art. 14.1)

a) Intervention logic showing interactions between priorities, focus areas and measures,

b) Common context (including RD related impact indicators), result/target, output indicators a performance review indicators,

c) Common evaluation questions,

8

Fiches for common context, impact, result/target and output indicatorGuidance on use and establishment of proxy indicators Guidance on the Indicator PlanGuidance on performance review and reserve

CEQ related to: →focus areas of RD priorities→other aspect of RDP (RDP synergies and TA)→Union level objectives

Elements of the CMES (Reg.808/2014, Art. 14.1)

d) Data collection, storage and transmission

e) Reporting on monitoring and evaluation activities

f) Evaluation plan

g) Ex ante, AIR 2017 and 2019 and ex post evaluation

h) Support to actors in M&E to fulfil their obligations

9

Common data sources: FADN, Eurostat, others (updated tables for CCI), II, RI.

Operations database and electronic transmission of monitoring data (OI)

Guidance on monitoring covering elements included in the AIR

Guidance on the Evaluation Plan preparation and implementation

Guidance on the ex ante evaluation of RDP 2014-2020

Other guidance documents 2014-2020

Intervention logic

Starting point for evaluation (ex ante, evaluation plan for evaluation during the programming period, ex post)

Composed of hierarchy of objectives (CAP, RD priority, focus areas, measures, operations)

Linked to financial allocations (inputs)

Linked to hierarchy of indicators

10

11

Europe 2020Europe 2020

CAP generalCAP generalobjectivesobjectives

Pillar I specificPillar I specificobjectivesobjectives

InstrumentsInstruments

Smart,

Resource efficiency flagship

sustainable and inclusive growth

CAP

Viable food production Sustainable management of natural resources and climate

action

Balanced territorial development

Contribute to farm incomes and limit

farm income variability in a

minimally trade distorting manner

Improve competitiveness of agricultural

sector and enhance share in food chain

Maintain market stability

Provide public goods (mostly

environmental) and pursue climate

change mitigation and adaptation

Foster resource efficiency through

innovation

Maintain a diverse

agriculture across the

EU

Intervention logic for Pillar I

Innovation Union flagship

Meet consumer

expectations

Direct payments Single CMO Horizontal and other instruments

• Basic payment

• Green payment

• Young farmers scheme

• Small farmers scheme

• Coupled support

• Support in areas facing natural constraints

• Market measures, including exceptional measures

• Producer organizations / Interbranch organisations

• School milk and fruit scheme

• Wine national support program and regulatory measures

• Cross compliance

• Quality policy

• Organic farming

• Promotion policy

• Research / EIP / FAS

Pillar I Intervention logic

12

Europe 2020

CAP generalObjectives

Pillar II specificObjectives (Priotities)

Smart, sustainable and inclusive growth

CAP

2. Competitivenessof all types of

agriculture and farm viability

3. Food chain organisation and risk management

4. Restoring, preserving and

enhancing ecosystems

5. Resource efficiencyand shift towards alow carbon and climate resilient economy

6. Social inclusion, poverty reduction

and economic development in rural areas

Intervention logic for Pillar II

1. Knowledge transfer and Innovation

Viable food production

Balanced territorial development

Sustainable management of natural resources and

climate action

Pillar II Intervention logic

1. Fostering knowledge transfer and innovation in agriculture, forestry and rural areas

RD Focus areas

1A Fostering innovation,

cooperation, and the development of the knowledge base in

rural areas

Relevant measures

Art. 36 C

o-operation

Art. 15 K

nowledge transfer

and information actions

Art. 16 A

dvisory services, farm

managem

ent and farm

relief services

1B Strengthening the links between agriculture,

food production and forestry and research

and innovation, including for the purpose of

improved environmental management and

performance

1C Fostering lifelong learning and

vocational training in the agricultural and

forestry sectors

RDP Intervention Logic

An Intervention logic will be drawn up for each RD priority showing their contributions to selected Focus Areas (including potential contribution of particular measure to several focus areas).

A basic intervention logic is proposed by the EC covering the most commonly expected combinations ( see following slides)

MAs have the flexibility to develop a specific intervention logic appropriate to their territory and its needs.

14

Evaluation

Ex ante evaluation (Art. 8.1 (a) and Art.77)

Evaluation plan (Art. 8.1 (a) )Common evaluation questions

Evaluation Questions (Pillar II)

Evaluation

questions

Policy objectiv

es

Indicators

Common Evaluation Questions: Horizontal and focus area evaluation questions:

define the focus of evaluations in relation to policy objectives, and;

help to demonstrate the progress, impact, achievements, effectiveness, efficiency and relevance of rural development policy.

Programme-specific evaluation questions

Common evaluation questions – 30

Focus area related – 18 questions Other aspects of RDP:

Operational performance - 1

Technical assistance - 1

National rural networks - 1

Horizontal evaluation questions EU 2020 - 5

CAP objectives - 4

4/5 June 2009

Challenges of the CMEF & Ongoing Evaluation

17

“Horizontal” Evaluation Questions

Linked to the overall policy objectives and other aspects of RDPs in order to demonstrate the achievements EU 2020 objectives

CAP objectives

RD cross-cutting priorities (environment, CC, innovation)

National Rural Networks

Technical Assistance

Operational performance (synergies)

18

“Horizontal” Evaluation Questions

Capture the contribution of the programme towards the overall policy objectives in terms of impacts.

Answered with the means of common impact indicators, common context indicators and complementary result indicators (and additional information when necessary)

Reporting in the AIR in 2019 and in the ex post evaluation.

19

Horizontal Evaluation QuestionsExample

20

To what extent has the RDP contributed to the CAP objective of fostering the competitiveness of agriculture?

JUDGEMENT CRITERIA • The agricultural

entrepreneurial income has increased

• The agricultural factor income has increased

• Agricultural productivity has increased

COMMON RD INDICATOR

• Agricultural entrepreneurial income

• Agricultural factor income

• Agricultural productivity

CAP objectiveFostering the competitiveness of agriculture

FA-related Evaluation Questions

Linked to the objectives of the Focus Areas in order to demonstrate the achievements towards the policy objectives

Capture the contribution of the interventions under each FA in terms of programme results

Answered with the means of result indicators (and additional information when necessary)

Reporting in the AIRs in 2017 and 2019 and in the ex post evaluation

21

FA-related Evaluation QuestionsExample22

Focus Area 4c: Preventing soil erosion and improving soil management

To what extent have RDP interventions supported the prevention of soil erosion and improvement of soil management?

JUDGEMENT CRITERIA

• Soil erosion has been prevented

• Soil management has improved

COMMON RD INDICATOR

• % of Agricultural land under management contracts to improve soil management (Result indicator)

• % of forestry land under management contracts to improve soil management (Result indicator)

ADDITIONAL INFORMATION

• Additional information on soil erosion of the land under management contracts

Common indicators 23

Common context indicators (45) refer to RDP context and include impact indicators

Financial (input) indicators refer to resources allocated to measures

Output indicators – PI (22) and PII (27) measure activities implemented within RDP/policy

Result indicators – PI (15) and PII complementary results (6)

measure direct and immediate effects of RDP/policy

Impact indicators (16) refer to benefits both at the level of measure or RDP/policy

Pro

gra

mm

e-sp

ecif

ic i

nd

icat

ors

(R

D)

Context indicators The Common Context Indicators (+ common impact

indicators) of 2014-2020: compulsory minimal set for the SWOT analysis, the RDP strategy and assessment of impacts 45 common context indicators

13 of 16 common impact indicators Socio economic; Sectorial; Environmental

Used in territorial description and SWOT, evaluations

Included in RDP via structured table (all required!) - the use of the common context indicators will be taken into account for the approval of RDPs by the EC

Base on the availble data, sources from EUROSTAT and other EU-level data sources, OR from national/regional sources. (Proxies may be used at regional level)

24

Context (Pillar II) and impact indicators

4/5 June 2009

Challenges of the CMEF & Ongoing Evaluation

25

Pillar I

EU price variability  

Agriculture trade balance  

Consumer price evolution of food products

Impact indicators

Cover both Pillar 1 and Pillar 2

Some are more relevant for Pillar 1 (e.g. trade related)

Some are more relevant for Pillar 2 (e.g. territorial development)

As far as possible existing datasets (EUROSTAT, Farm Structure Survey, FADN etc.) available at EU, national and/or regional level will be used for quantification.

26

Results indicators

25 result indicators

Show direct achievements of policy

Linked to Focus Areas of P2-5

Mix of:

targets (19) - monitoring

complementary result indicators (6) - evaluation

Requirement to flag operations contributing to Focus Areas with complementary result indicators

4/5 June 2009

Challenges of the CMEF & Ongoing Evaluation

27

Target indicatorsAt least one quantifiable target indicator is required for each Focus Area.

Mostly based on result indicators (19),

Some are closer to output indicators – monitoring (e.g. Priority 1).

Show direct achievements of policy

Linked to Focus Areas of P2-5

The target indicators will be reported on annually in the AIRs using:

Direct monitoring data,

Estimates based on coefficients supplied in the guidance (e.g. to estimate the production of renewable energy from new investments).

In few cases the calculation of the target indicators will be conducted by the evaluator when assessing the achievements of RDP (e.g. water and energy savings)

28

Output indicators

26 indicators at measure level:

Output indicators - for all measures (total public expenditures) Output indicators linked to several measures (area under…) Output indicators linked to specific measures (support for

Leader start up)• Only a selection in the Indicator plan (Planned Outputs to be

quantified)• New indicators for NRN, Risk Management….

29

Programme Specific Indicators

(Article 50(2) 1303/2013; draft RD IA Annex I Part 1 Point 4(a)(i))

Design in SMART way,

Shall:

provide additional info specific to RDP territory e.g.

Definition of rural area

Forestry sector

describe issues where common data lacking e.g.

Innovation

Short supply chains

Local markets

support and justify particular interventions e.g.

Thematic sub-programmes

Programme specific Focus Areas

Methodological approach (PII)31

Setting up the M&E system

- Evaluation Plan- Governance - ToR- Prepare EQ and

indicators

Preparing and Structuring- Intervention logic- Establish methodology- Identify indicators

Observing - Collection of data and qualitative information

Analysing- Process and synthetise data and information- Calculating Net effects

Judging- Conduct

Assessment of impacts

- Answer EQ- Conc&Rec

Reporting and

disseminating

Reporting

“Standard” AIR – every year from 2016

Enhanced AIR 2017

Enhanced AIR 2019

“Standard” AIR – every year from 2016

(Art 75 1305/2013 and 50 1303/2013)

Key information on implementation of the programme and its priorities (Financial and monitoring)

Issues which affect the performance of the programme and the measures taken

Steps taken to implement technical assistance and programme publicity requirements

Actions taken to fulfil ex ante conditionalities (in 2017 and in 2016 where relevant)

+ Annex template on implementation of the financial instruments

Content of "standard" AIR (reporting on evaluation activities)

EP modifications

Evaluation activities undertaken

Activities related to provision and management of data

List of completed evaluations

Summary of completed evaluations

Communication activities

Follow-up of evaluation results

Enhanced AIR 2017

Information from evaluation activities on: Reporting and quantification of programme

achievements, in particular through assessment of the complementary result indicators, and relevant evaluation questions.

Description of implementation of sub-programmes Implementation of actions to take into account the

principles set out in art 6, 7 and 8 CPR (Art. 50 CPR)a) Promotion of equality between men and women and non-discrimination b) Sustainable development (Art. 8 CPR)c) The role of the partners referred to in Article 5 CPR in the implementation of the programme and preparation of the progress report

Enhanced AIR 2019

Information from evaluation activities on:

Reporting and quantification of programme achievements, in particular through assessment of the complementary result indicators, and relevant evaluation questions.

Reporting on interim impact of the RDP: contribution to programme and EU strategy and objectives, in particular through assessment of the programme's net contribution to changes in CAP impact indicator values, and relevant evaluation questions.

Progress made in ensuring integrated approach

37

Thank you for your attention!