From Building Evaluation Capacity to Supporting Evaluation Capacity

Development: Key Conclusions of an exploratory study conducted in DRC, Niger and South Africa

Dr. Michele TarsillaOPEV Week - African Development Bank

December 6, 2012E-mail: mitarsi@hotmail.com

Presentation OutlinePresentation Outline

Background on Evaluation Capacity Building (ECB) and Evaluation Capacity Development (ECD)

Two key research questions

Conclusions-ECD targeting-New ECD definition/conceptualization

Key ECB et ECD IssuesKey ECB et ECD Issues1) Lack of both a consensus on and clarity

of Evaluation Capacity Development (ECD) definition and goals and need for more empirical research

2) Biased targeting of past ECD programs aimed at strengthening in-country evaluation capacity

3) Definitions are key and so are strategies to make ECD processes more inclusive. In addition, understanding and acting upon processes (outside the training venues) is of utmost importance.

First Research First Research QuestionQuestion

To what extent is

Evaluation Capacity

Development (ECD)

distinct from

Evaluation Capacity

Building (ECB) in

international

development

contexts?

What is Capacity?What is Capacity?

Capacity as a latent attribute:Capacity is a potential state. It is

elusive and transient. Performance, in contrast, is about execution and implementation or the result of the application/use of capacity (Morgan, 2006, p. 6)

How do ECB and ECD How do ECB and ECD relate?relate?

69% (n=51) of the 75 respondents who were asked about the relationship between ECB and ECD perceived that there was a difference between ECB and ECD

69% (n=51) of the 75 respondents who were asked what they associated ECB with the most, answered trainings

(64%) (n=48) of the 75 respondents who were asked about ECB sustainability stated that ECB was no longer effective or sustainable

Nearly 59% (n=44) of the 75 respondents who were asked what they thought about the concept of ECB stated that the idea of “building” capacity did not do justice to the fact that there is already capacity in country.

Evaluation Capacity Evaluation Capacity BuildingBuilding

According to respondents, ECB key objectives were:

-To enhance the data collection and reporting skills of donor-funded projects: 73% of respondents (n=65) of the 90 in-country practitioners;

-To improve the project performance and attain the expected project results: 54% of respondents (n=49) of the 90 in-country practitioners;

-To promote the use of the Logical framework: 77% (n=69) of respondents;

Evaluation Capacity Evaluation Capacity DevelopmentDevelopment

According to respondents, ECD primary objectives were:-To create an in-country sustainable evaluative culture: 73% (n=65) of interviewed field practitioners;-To reduce countries’ dependence/reliance on external technical support: 67% (n=60) of interviewed in-country practitioners;-To promote the use of evaluation within national governments: 59% (n=53) of interviewed in-country practitioners; “Unlike ECB that has a more individual-focused approach, ECD is about people and systems” (South Africa government official)

Second Research Second Research QuestionQuestion

What are the

key criteria that

need to be taken

into account in

order to enhance

ECD targeting?

SSpherephere

Role

Function

SFAR Framework

Actors

SFAR FrameworkSFAR Framework

SFAR in NigerSFAR in Niger

ConclusionsConclusions

Key ConclusionsKey Conclusions

New definitions of ECB and ECD

Key variables to evaluate the effectiveness of ECD programming and processes

New ECB DefinitionNew ECB DefinitionA necessary (but not sufficient) condition

for ECD to take place. ECB mainly consists of a vast array of trainings and coaching activities (some of which are short-term in nature) aimed at building capacity

especially where capacity is either very low or thought not be in place yet,

among a discrete number of individuals working either for or within organizations and/or institutions that develop, commission, manage, conduct and/or use evaluation.

ECB is normally implemented in either one or two of the three ECD levels

ECD Continuum: ECD Continuum: Where does ECB lie? Where does ECB lie?

ECD Continuum: ECD Continuum: When ECB Gets CreativeWhen ECB Gets Creative

New ECD Definition (I)New ECD Definition (I)ECD is a process consisting in both the

integrated enhancement and maintenance over time of:

-individuals’ knowledge, skills and attitudes; -organizations’ capabilities; and -institutions’ readiness,

towards contextually relevant planning, management, implementation, and use of evaluation at any level-global, regional, national or sub-national.

ECD Continuum: ECD Continuum: Where does ECD lie? Where does ECD lie?

New ECD Definition (II)New ECD Definition (II)More specifically, ECD aims at both individual and collective transformational learning in the pursuit of three primary goals: -strengthening the technical quality and ownership of evaluation processes; -enhancing the local authenticity and cultural appropriateness of evaluation approaches, methods and tools used in-country; -increasing the use of evaluation as a way to improve development interventions in a variety of sectors.

New ECD Definition (III)New ECD Definition (III)In order for ECD to be successful, it is

critical that ECD-savvy strategies be implemented either in a simultaneous or intentionally sequenced fashion;

ECD-savvy strategies are specifically aimed at promoting conditions that enable ECD among a variety of actors operating in two different spheres (within and outside of national governments) and characterized by different functions (operational and policy- or decision-making) and roles (consumers and providers of evaluation).

New ECD Definition (IV)New ECD Definition (IV)In an effort to enhance ECD

ownership and sustainability, it is relevant that sphere-crossing entities, either national (e.g., Voluntary Organizations Promoting Evaluation or VOPEs) or multi-national (regional evaluation associations), actively contribute to ECD-savvy strategies

Key PrinciplesKey Principles Evaluation of processes – not just performance-

inherent to ECD stakeholders (e.g. inter-organizational dynamics, availability of “learning space” within an institution, resilience at time of crisis) is key to identifying a priori ECD enabling factors or barriers;

Assessing the level of ECD needs, interests and motivation across spheres and across functions within spheres is critical to customize and sequence ECD programs;

Gauging the quality and the degree of ECD targeting’s inclusiveness is instrumental in the identification of any possible inequity (and strategy available to address it) in ECD programming;

Dissemination of details on the budget resources available for and effectively spent for the implementation of ECD programming is key to cost-benefit and cost-effectiveness analyses;

Assessing the feasibility of ECD success given the established timeframe is relevant

Evaluating ECD Evaluating ECD Effectiveness (I) Effectiveness (I)

BEFORE THE IMPLEMENTATION OF AN ECD PROGRAM:1.Conduct an evaluation capacity needs assessment within each sphere;2.Conduct a diagnostic of evaluation processes within each sphere (use secondary data as appropriate) 3.Gauge the knowledge level of Monitoring and familiarity with Results-based management (RBM)

DURING or RIGHT AFTER IMPLEMENTATION:1.Assess short-term results (such as, increase in knowledge and development of technical skills) produced by ECD activities for each of the three levels;

Evaluating ECD Evaluating ECD Effectiveness (II) Effectiveness (II)

A LITTLE AFTER IMPLEMENTATION:

1.Start assessing systematically medium- and longer-term results produced by ECD activities implemented at the organizational and institutional levels;

2.Conduct individual or group follow-up interviews and make the best use of online tools to foster reflections and conversations on the lessons/challenges resulting from ECD programs

THROUGH ALL ECD PROGRAMMING PHASES:

1. Assessing the internal processes (e.g., type and quality of interactions among actors in different spheres and with different functions as well as the degree to which the targeted individuals could be both providers and consumers at once);

2.Monitor the assumption underlying your Theory of Change;

3.Be very systematic in your assessment of the mediating factors (both those included in your theory of change and others that you might identify during implementation)

Study LimitationsStudy LimitationsThe applicability of the

conclusions might be limited;The validity of the framework

could only enhanced by testing it to a larger number of countries;

The ideas collected among practitioners interviewed in the course of data collection may not be representative of the whole membership;

Thank you!Thank you!

Contact:Michele Tarsilla, Ph.D. International Evaluation Advisor and Capacity Development Specialist

E-mail: mitarsi@hotmail.com