agrir

egio

nie

urop

a

Exploring the perspectives of a mixed case study approach for the evaluation

of the EU Rural Development Policy 2007-2013

Ida Terluin and Petra BerkhoutAgricultural Economics Research Institute LEI

122nd European Association of Agricultural Economists Seminar

Evidence-Based Agricultural and Rural Policy MakingMethodological and Empirical Challenges of Policy Evaluation

February 17th – 18th, 2011, Ancona (Italy)

associazioneAlessandroBartola studi e ricerche di economia e di politica agraria

Centro Studi Sulle Politiche Economiche, Rurali e AmbientaliUniversità Politecnica delle Marche

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Common Monitoring and Evaluation Framework (CMEF)

CMEF is designed by the EU Commission for the

evaluation of the EU Rural Development Policy 2007-

2013

Comprehensive approach: about 160 indicators and 140

common evaluation questions

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Concerns on the CMEF

large number of indicators is experienced as a cumbersome

requirement of Brussels; data are not always available

simplification of the CAP: wish to have a simpler and more popular kind

of evaluation framework

evaluation questions are not always relevant in Member States/regions

emphasis on what has happened and not how or why it happened

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Aim of our paper

To explore whether there is an alternative approach to

the CMEF for the evaluation of the EU Rural

Development Policy 2007-2013 and to asssess the

perspectives of this alternative for EU wide use

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Plan of this presentation

1. Comparative analysis of evaluation methods of rural development

policy

2. Identification of an alternative approach for the CMEF

3. Testing of this alternative approach in the midterm evaluation of the

RDP 2007-2013 in The Netherlands

4. Assessment of the perspectives of this alternative approach for EU

wide use

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Comparative analysis of evaluation methods of rural development policy

Literature study: 22 recent evaluation methods

Classification of these methods according to applied methodology

into five goups:

1. the CMEF type approach: hierarchy of indicators combined with evaluation  questions

2. the tally approach: counting whether a quantified objective has been achieved

3. the econometric approach: use of econometric methods

4. the modelling approach: use of models

5. the mixed case study approach: use of quantitative and qualitative analyses, often in case studies

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Main properties of evaluation methodsApproach

CMEF type

tally econo-metric

model-ling

mixed case study

What happened in quantitative terms

x x x x

What happened in qualitative terms x

How and why did it happen x

Unintended effects of policy x

Why actors participate in policy x

Easy to apply for evaluator x x

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Identification of an alternative approach to the CMEF

Starting point for identification: different properties of the

groups of evaluation methods

Given differences of CMEF and mixed case study

approach, this last approach might be a promising

alternative for the CMEF

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

The mixed case study approach

analysis consists of different stages, in which different methods are

applied

by combining evidence from these stages, it is tried to find exploring

patterns

approach has been applied for the ex post evaluation of the EU Rural

Development Policy 2000-2006 in Wales by Midmore et al. (2008):

1. analysis of secondary data of the case study region

2. in-depth interviews with representatives of different interest groups

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Design of a protocol with instructions for the various steps in the evaluation

1. Analysis of the baseline situation in the case study region

2. Analysis of the objectives of the EU Rural Development

Programme (RDP) 2007-2013 in the case study region

3. Analysis of the financial input and the output for each rural

development measure

4. Questions for in-depth interviews with representatives of the

different interest groups

5. A template for writing a report

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Testing alternative in midterm evaluation RDP in 2 Dutch provinces:(1) Zeeland and (2) Gelderland

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Experiences in case study in Zeeland (1)

Baseline situation in the case study region: easy to describe, as data

and literature were readily available

Objectives of the RDP: difficult to assess, due to complex way the

RDP is implemented in The Netherlands

– there is one RDP for the whole country

– national government is responsible for axes 1 and 2

– province is responsible for axes 3 and 4 and has integrated them

with other rural policies

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Experiences in case study in Zeeland (2)

Data on financial input and output indicators for each rural development

measure are collected by two institutes, which have different data systems

Data for input and output indicators for measures under axes 1 and 2 are

only available at national level, and it is very time-consuming to extract

regional data

Data on result and impact indicators in Zeeland are not collected

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Experiences in case study in Zeeland (3)

Progress in the absorption of the EAFRD budget 2007-

2009 in Zeeland:

Group 3 focuses on tourism activities, basic services,

village renewal and rural heritage, measures which fit

rather well into the rural development needs in Zeeland

1 Axes 1 and 2 no information

2 Measures 311, 312, 41 moderate

3 Measures 313, 321, 322 and 323

substantial

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Experiences in case study in Zeeland (4)

According to the six interviewed persons:

the landscape and quality of life have been improved in qualitative

terms by the RDP

these projects fit in the long tourist tradition in Zeeland as well as

the need to maintain or improve the supply of basic services in rural

areas

economic diversification has been boosted

they had no idea of the impact of the measures of axis 1 on the

development of the agricultural sector

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Experiences in case study in Zeeland (5)

actors participate in the RDP measures as this generates funds for

their projects

most of the projects would also have been conducted without

EAFRD financing, but often at a smaller scale

the requirement of 50% national cofinancing for each project is not

always feasible

interviewees appreciated the easy communication and the

openness of the province of Zeeland, which stimulates the

implementation of the measures of axes 3 and 4

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Concluding remarks (1)

1. Based on a comparative analysis of 22 evaluation methods of rural

development policy, we have identified the mixed case study approach

als an alternative to the CMEF

2. The results of testign this alternative in the province of Zeeland are

promising: the analysis of inputs and outputs of the measures in

combination with interviews provided sufficient information to get a good

overview of the performance of the RDP

agrir

egio

nie

urop

a

122nd EAAE Seminar, February 17th – 18th , 2011, Ancona (Italy)

Concluding remarks (2)

3. The interviews yielded useful insights into processes within the

region

4. When experiences with the mixed case study approach in other

regions are also satisfying, then it could be considered to use it EU

wide as an alternative to the CMEF