4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

1

Common monitoring and evaluation framework

Jela Tvrdonova, 2010

Strategic approach to rural developmentFocus on limited number of objectivesCompetitiveness – Environment – Quality of life

Axes connected with strategic objectivesStrong and dynamic agri-food sectorAgriculture and forestry with high added valueEmployment and growth in rural areasImprovement of the governance in rural areas and

mobilisation of the endogenous potential

Common approach to monitoring and evaluationBased on: Exact definition of objectives in Regulation, Strategic

Guidelines for RD and rural development programs EU Strategic monitoring a national strategies Definition of baseline indicators at the program start

period Suitable combination of output, result and impact

indicators Single framework for all program intervetion

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

4

Content of the CMEF

CMEF – Common monitoring and evaluation framework – Handbook, Annexes, Guidance notes

Indicators and their role in the intervetion logic of rural development programs

Common and additional/program specific indicators

CMEF orientation in relation to indicators The role of indicators in particular evaluation

tasks

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

5

CMEF – legal base

EC Reg. 1698/2005 and 1974/2006 and its Annex VIII

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

6

Monitoring and evaluation

Monitoring – measurement of inmediate outcomes and results at measure and axis level

Evaluation - measurement of long-term and rural areas effects/impacts of intervention, comparissons and lessons learned for next interventions

Monitoring

On-going process which monitors the gradual implementation of the program at the level of financial inpiuts, physical outputs and axes results

It subject of annual reporting on the program implementation

Instruments – common and additional indicators of output and results,

Strategic monitoring

Since 2010 and every other two years Focus of the EC on the progress of

national strategic plans and its objectives implementation

Assesment of the contribution of national plans to the EU Rural Development Strategy

Evaluation

On-going evaluation is based on CMEF and country´s own methodology to carry on the evaluation during the program implementation

Process based on annual reporting of result/impact indicators including periodical exercises:

Ex – ante: evaluation of relevance of the planned intervention and optimizing of the RDP budget

Mid-term and ex-post: for the assessing the progress related to the specific and overall/program objectives,

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

10

Monitoring and evaluation in program logic

Good program is basic ground for the high quality of monitoring and evaluation

The evaluation and the monitoring strats with the program intervention logic

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

11

Intervention logic of RDP

Needs

Change observed in reality during and/or

after intervention

Envisioned change before intervention

Impact

Result

Output

Inputs of the RDPMeasures

Overall/horizontal objectives of RDP

Specific/Axis objectives of the RDP

Operational/measure objectives of the RDP

Change

Overall baseline socio-economic and environmental situation of rural areas, at baseline period

Development context - all intervening

factors

Hierarchy o objectives and indicators 5 types of indicators coresponding with the hierarchy of

objectives Hierarchy of objectives: - instrument helping to show

how local activities contribute to the overall objectives, It is composed of:

- operational objectives (measure level)- specific objectives (Axes/priority areas level)- overall – cross cutting/strategic/horizontal objectives (at the program level)

Indicators copy objectives in the hieararchy

Indicators in intervention logic of RDP

Hierarchy of indicators

Hierarchy of objectives

Impact ndicator

Result indicator

Output indicato

Input indicators

Measures

Overall/horizontal objectives of RDP

Specific/Axis objectives of the

RDP

Operational/measure objectives of the

RDP

Context related baseline indicators

Objectives related baseline indicators

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

14

Role of indicators

Indicators – used as tools to assess how far the expected

objectives have been achieved by measures or whole programmes

– should be specific, measurable, available / achievable in a cost effective way, relevant for the programme, and available in a timely manner (SMART)

– Indicators can not always be filled with quantitative statistical data; in some cases, indicators might also include qualitative assessments or logical assumptions

The quality of indicators - SMART approachS – simple, specific

M – measurable - data exist

A – achievable – available – at low cost

R – relevant – selected well for given program

T - timely oriented

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

16

Types of indicators

Baseline indicators: they relate to general socio-economic context of the programme area (context-related baseline indicators) and to the state of the economic, social or environmental situation in direct relation with the wider objectives of the programme (objectives-related baseline indicators)

Financial execution (input) indicators: they refer to the budget or other resources allocated to the programmes

Output indicators: measure activities directly realized within programmes

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

17

Types of indicators

Result indicators: measure the direct and immediate effects of the intervention and provide information on changes that have taken place

Impact indicators: refer to the benefits of the programme both at the level of the intervention but also more generally in the programme area. They are linked to the wider objectives of the programme

The process

Inputs

Activities

Result

Ouputs

Impact

Monitoring and Evaluation

process

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

19

Common indicators

A common set of baseline, output, result, and impact indicators for the RDPs (Art. 62 Reg. 1974/2006) “shall form the Common Monitoring and Evaluation Framework (CMEF)’’ (Annex VIII lists the common indicators)

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

20

Additional indicators

Since common indicators may not fully capture all effects of programme activities, it is necessary to define additional indicators within the programmes (see Guidance notes A & K)

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

21

Why additional indicators?

The CMEF provides the Common Framework to be applied throughout the EU, thus ensuring comparability

The specificities of each single programme cannot be fully reflected by this Common Framework

It is therefore important to complement the Common Framework by additional indicators in order to capture the full range of intended effects of a given programme

As general rule, a thorough analysis of the programme intervention logic can drive the choice of relevant additional indicators

Considering the limited number of common impact indicators and their broad scope, additional indicators are essential to overcome attribution gaps

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

22

CMEF structure

HandbookAnnex 1: Guidance notes - A: Choice and use of indicators, B:

On-going evaluation , C: Mid-term evaluationAnnex 2: Guidance notes – D: Hierarchy of objectives, E –

Measure FichesAnnex 3: Guidance notes – F – K, Common indicators FichesAnnex 4 Guidance J – O Other guidances

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

23

CMEF guidance on indicators

CMEF provides guidance for MAs in setting up indicators at each level of intervention and baseline indicators

Annex 3 of the CMEF Handbook provides detailed description of all common indicators (Guidance notes F – K):

– F: COMMON INDICATOR LIST (overview of all common indicators)

– G. BASELINE INDICATOR FICHES (detailed description of indicators)

– H. OUTPUT INDICATOR FICHES

– I. RESULT INDICATOR FICHES

– J. IMPACT INDICATOR FICHES

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

24

Indicator Fiches

Each indicator fiche contains the following elements: Type of indicator Related measures Measure Codes Definition of the indicator Subdivision Unit of measurement Level of collection Responsible actor for collection Collection method/good practice Sources Registration frequency

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

25

Measure Fiches

Guidance on practical use of indicators in monitoring and evaluation of RDPs

Guidance note E provides the following on use of indicators within each particular measure:

– Measure Code– Rationale of the measure– Target group– Target area– Common indicators – Link rationale of the measure and indicators– Evaluation questions

4/5 June 2009 Challenges of the CMEF & Ongoing Evaluation

26

Setting up the evaluation framework at the stage of Ex-ante and use of indicators

Evaluation Question Indicators

Evaluation questions – cross-cutting

Evaluation questions Axis/sector specific

Output indicators

Impact indicators

Result indicators Axis based baselines, objective related, contextual

Baselines and their quantification,

Target levels

Horizontal baselines, objective related, contextual

RDP objectives

Overall strategic objectives

Axis specific objectives

Measure/operational objectives

Baseline values

of output indicators

Targets for impact indicators

Targets for results indicatiors

Targets for output indicators

Common and program spec. Eval. questions