PERSPECTIVES ON IMPACT EVALUATIONApproaches to Assessing Development Effectiveness

CAIRO, March - April 2009

Ignacio PardoUniversidad de la República, Uruguay

2005 - 2008

Mixed - Methods Impact Evaluation

2009 -2010

2002 Crisis Poverty and indigence Long – term unemployment Very few social programs for the

most poor and unemployed in the past

Ministry of Social Development, created in 2005

Evaluation: accountability, feedback

Conditional Cash Transfer program:“Ingreso

Ciudadano” Adult Literacy program Educative interventions Housing programs … Job training program:

TRABAJO POR URUGUAY

Increasing importance of mixed methods research in many disciplines

Evaluation: early use of mixed methods in evaluation, as in other “practical” fields (education, for example)

Recent systematization of mixed – methods research (Greene, 2007; Tashakkori and Teddlie, 2009)

(+ Monitoring)

QUANTITATIVE + QUALITATIVE APPROACH(Mixed Methods? We know why… But… How? When?)

QUAN(Survey)

QUAL(Focus Groups,

In-Depth Interviews)

Plan Level (PANES)

Impact (mainly, CCT)

Program Level (Trabajo Por

Uruguay)Outputs / Outcomes

Process Outputs / Outcomes

“Qual” Objectives

Meta Analysiis of

data

Survey

“Quan” Objectives

In depth Interviews + Focus Groups

“Qual” Analysis“Quan” Analysis

Triangulation

Complementarity

Development 

Initiation

Expansion 

Concurrent Sequential

Equal QUAL + QUANQUAL -> QUANQUAN -> QUAL

DominantQUAL + quan

QUAN + qual

QUAL -> quanQual -> QUAN

QUAN -> qualQuan -> QUAL

Usual approach:

Community self – analysis, implementation, PROCESS EVALUATION:

Qualitative tools

Outputs, outcomes, IMPACT EVALUATION:

Quantitative tools

Plan level: PLAN DE EQUIDAD

Program level: URUGUAY TRABAJA

Similar to “Trabajo por Uruguay” Evaluation design:

• NEW DESIGN? YES • MIXED – METHODS APPROACH? YES• IMPACT EVALUATION? LUCKILY• QUALITATIVE IMPACT EVALUATION? PROBABLY

“RCTs are not always best for determining causality and can be misleading. RCTs examine a limited number of isolated factors that are neither limited nor isolated in natural settings. The complex nature of causality and the multitude of actual influences on outcomes render RCTs less capable of discovering causality than designs sensitive to local culture and conditions and open to unanticipated causal factors.” (American Evaluation Association response to US Department of Education, 2003)

“RCTs are not always best for determining causality and can be misleading. RCTs examine a limited number of isolated factors that are neither limited nor isolated in natural settings. The complex nature of causality and the multitude of actual influences on outcomes render RCTs sometimes less capable of discovering causality than designs sensitive to local culture and conditions and open to unanticipated causal factors.” (American Evaluation Association response to US Department of Education, 2003, and me)

Increasing number of experts answering yes and trying to develop a more complete answer on how

RCT is not the ideal approach in every case. It is approapiate in certain cases (few variables, stable situation, context not affecting variables, clear expected outcome…) (Perrin, Tuesday).

Lipsey – Scriven debate Qualitative Impact Evaluation analysis

techniques suggested, as the Qualitative Comparative Analysis (QCA) (O’Reilly, 2008)

“In many instances, it is difficult to know in advance the various factors impinging on outcomes and in the absence of that knowledge, one cannot empirically isolate the role of the intervention relative to other (extraneous) factors. In this context, information on processes, institutions, and on perspectives of participants (and non-participants) themselves can be very helpful. These approaches are also well suited to explain in-depth the reasons for, and character of, critical incidents and events, and the dynamics or causality of such events when they are linked into sequences or processes” (Ezemenari et al,1999)

Physical causality will often have a much more inmediate and compelling “presence” than relations that are only of the “if not X” variety. (…) And it is eminently available to the qualitative research approach (that is) well suited to establish causality (Mohr, 1995)

Interpretation of causality > causal logic

Synthesis of quantitative and qualitative methods will be the best approach to causality (Davidson, 2005)

“Unpacking the complexity of the change process” (Sridharam, yesterday), looking as precipitating causes, amplyfing causes, causes of vulnerability to change. And finding the best approach to each of this kind of causation. Thinking of evidence as in a trial, looking for traces…

QUAL methods can answer the important question: What Makes What Works Work? (Scriven, 2008)

Impact evaluation lies on counterfactual causality. This is usually regarded as the most robust approach. For good reasons. But it is only one.

At the same time, we can take into account other types of causality. Specifically, physical cause (modus operandi technique, forensic analysis, direct observation)

Reasons as causes (Berge, 2007) “The qualitative method do not rely on any

inference about the counterfactual: it relies on establishing with high probability a physical cause” (Berge, 2007)

Causal relations can be observed (realistic framework) and depend on context, which allows qualitative explanation to get into attribution analysis (Maxell, 2004)

QUAN methods can tell us what and where meanwhile QUAL methods are better tools to answer why and how.

• That is why the latter can be maybe the main tool in Impact Evaluation, in a process of “creative experimentalism” (Prowse, Tuesday)

• Life History method can be a powerful QUAL tool, generating QUAN and QUAL data, besides visual data on trajectories. And providing a specific approach on causality

In the next months, an evaluation design will be elaborated for “Uruguay Trabaja”

Mixed – methods evaluation may be taked one step further• Mixing methods at earlier steps; using the mixing for

more purposes The concept of Impact Evaluation can

probably be expanded in orden to give a better picture of the causal relations that will take place in the context of the program

Work in progress…