Impact Eval and Project Cycle

8/7/2019 Impact Eval and Project Cycle

1/37

Impact Evaluation andImpact Evaluation and

the Project Cyclethe Project CycleArianna LegoviniArianna Legovini

PREM WEEK May 2, 2006PREM WEEK May 2, 2006This presentation is based on work by theThis presentation is based on work by the

Thematic Group on Impact EvaluationThematic Group on Impact Evaluation


2/37

Objective of the presentationObjective of the presentation

Walk you through what it takes to do anWalk you through what it takes to do an

impact evaluation for your project fromimpact evaluation for your project from

Identification to ICRIdentification to ICR

Persuade you that impact evaluation willPersuade you that impact evaluation will

add value to your projectadd value to your project


3/37

We will talk aboutWe will talk about

Objective of the evaluationObjective of the evaluation

General PrinciplesGeneral Principles

Evaluation activitiesEvaluation activities the core issues forthe core issues forevaluation design and implementation,evaluation design and implementation,

andand

Housekeeping activitiesHousekeeping activitiesprocedural,procedural,administrative and financial managementadministrative and financial management

issuesissues


4/37

Why do an impact evaluation ofWhy do an impact evaluation of

your project?your project?Provide a sound basis for policy developmentProvide a sound basis for policy development

Measure the impact of the project on intended andMeasure the impact of the project on intended and

unintended outcomes for:unintended outcomes for: Making budget decisions and reallocating resources (fiscalMaking budget decisions and reallocating resources (fiscal

accountability)accountability)

Scaling up (provisos apply)Scaling up (provisos apply)

Measure the relative effectiveness of alternativesMeasure the relative effectiveness of alternatives(modes of delivery, packages, pricing schemes) to:(modes of delivery, packages, pricing schemes) to: Modify project features overtime (managing by results)Modify project features overtime (managing by results)

Inform future project designsInform future project designs


5/37

Some general principlesSome general principles

Government ownershipGovernment ownershipwhat matters iswhat matters is

institutional buyinstitutional buy--inin

Relevance and applicabilityRelevance and applicabilityasking theasking theright questionsright questions

Flexibility and adaptabilityFlexibility and adaptability

Horizon mattersHorizon matters


6/37

Country ownershipCountry ownership Ensure Government involvement at all stages toEnsure Government involvement at all stages to

build institutional capacity and a culture ofbuild institutional capacity and a culture of

managingmanaging--byby--results.results.

Agree on a dissemination plan to maximize use ofAgree on a dissemination plan to maximize use ofresults for policy development.results for policy development.

Identify entry points in project and policy cyclesIdentify entry points in project and policy cycles

midpoint and closing, for project;midpoint and closing, for project;

sector reporting, CGs, MTEF, budget, for policysector reporting, CGs, MTEF, budget, for policy

Use partnerships with local academics to buildUse partnerships with local academics to build

local capacity for impact evaluation.local capacity for impact evaluation.

Example, Kenya educationExample, Kenya education


7/37

Relevance and ApplicabilityRelevance and Applicability

For an evaluation to be relevant, it must beFor an evaluation to be relevant, it must bedesigned to respond to the policy questions thatdesigned to respond to the policy questions thatare of importance to the clients.are of importance to the clients.

Clarifying early what it is that the client wants toClarifying early what it is that the client wants tolearn and designing the evaluation to that endlearn and designing the evaluation to that endwill go some way to ensure that thewill go some way to ensure that therecommendations of the evaluation will feed intorecommendations of the evaluation will feed into

policy making.policy making. Example, devolution and decentralization inExample, devolution and decentralization in

ZambiaZambia


8/37

Flexibility and adaptabilityFlexibility and adaptability

The evaluation must be tailored to the specificThe evaluation must be tailored to the specificproject and adapted to the specific institutionalproject and adapted to the specific institutionalcontext.context.

The project design must be flexible to secure ourThe project design must be flexible to secure our

ability to learn in a structured manner, feedability to learn in a structured manner, feedevaluation results back into the project andevaluation results back into the project andchange the project midchange the project mid--course to improvecourse to improveproject end results.project end results.

Example, Ethiopia energyExample, Ethiopia energy

This is an important point: In the past projectsThis is an important point: In the past projectshave been penalized for affecting midhave been penalized for affecting mid--coursecoursechanges in project design. Now we want tochanges in project design. Now we want to

make change part of the project design.make change part of the project design.


9/37

Horizon mattersHorizon matters The time it takes to achieve results is an importantThe time it takes to achieve results is an important

consideration for timing the evaluation. Conversely, theconsideration for timing the evaluation. Conversely, thetiming of the evaluation will determine what outcomestiming of the evaluation will determine what outcomesshould be focused on.should be focused on. Early evaluations should focus on outcomes that are quick toEarly evaluations should focus on outcomes that are quick to

show changeshow change For longFor long--term outcomes, evaluations may need to span beyondterm outcomes, evaluations may need to span beyond

project cycle. Example, Madagascar early childhoodproject cycle. Example, Madagascar early childhooddevelopmentdevelopment

Think through how things are expected to change overThink through how things are expected to change over

time and focus on what is within the time horizon for thetime and focus on what is within the time horizon for theevaluationevaluation

Do not confuse the importance of an outcome with the timeDo not confuse the importance of an outcome with the timeit takes for it to changeit takes for it to changesome important outcomes aresome important outcomes are

obtained instantaneously !obtained instantaneously !


10/37

Identification to PCNIdentification to PCN


11/37

Get an Early StartGet an Early StartHow do you get started?How do you get started?

Get help and access to resources: contactGet help and access to resources: contactperson in your region or sector responsible forperson in your region or sector responsible forimpact evaluation and/or Thematic Group onimpact evaluation and/or Thematic Group on

Impact Evaluation and HDN CEO

Impact Evaluation and HDN CEO

Define the timing for the various steps of theDefine the timing for the various steps of theevaluation to ensure you have enough lead timeevaluation to ensure you have enough lead timefor preparatory activities (e.g. baseline goes tofor preparatory activities (e.g. baseline goes tothe field before program activities start)the field before program activities start)

The evaluation will require support from policyThe evaluation will require support from policy--makers: start building and maintainingmakers: start building and maintainingconstituents, dialogue with relevant part ofconstituents, dialogue with relevant part ofgovernment, build a broad base of support,government, build a broad base of support,include stakeholdersinclude stakeholders


12/37

Build the TeamBuild the Team Select impact evaluation team and defineSelect impact evaluation team and define

responsibilities of:responsibilities of: program managers (government),program managers (government),

project team, and other donors,project team, and other donors,

lead researcher (impact evaluation specialist),lead researcher (impact evaluation specialist),

local research team, andlocal research team, and data collection agency or firmdata collection agency or firm

Selection of lead researcher is critical for ensuringSelection of lead researcher is critical for ensuringquality of product, and so is the capacity of the dataquality of product, and so is the capacity of the data

collection agencycollection agency

Partner with local researchers and researchPartner with local researchers and researchinstitutes to build local research capacityinstitutes to build local research capacity


13/37

Shift ParadigmShift Paradigm

From a project design based on we know whats bestFrom a project design based on we know whats best

To project design based on the notion that we can learnTo project design based on the notion that we can learnwhats best in this context, and adapt to new knowledge aswhats best in this context, and adapt to new knowledge asneededneeded

Work iteratively:Work iteratively: Discuss what the team knows and what it needs to learnDiscuss what the team knows and what it needs to learnthethe

questions for the evaluationquestions for the evaluationto deliver on project objectivesto deliver on project objectives

Discuss translating this into a feasible project designDiscuss translating this into a feasible project design

Figure out what questions can feasibly be addressedFigure out what questions can feasibly be addressed

Housekeeping: Include these first thoughts in a paragraph in theHousekeeping: Include these first thoughts in a paragraph in thePCNPCN


14/37

Example: Incorporate learning in anExample: Incorporate learning in an

energy access in Ethiopiaenergy access in Ethiopia One component of the project will distribute CompactOne component of the project will distribute CompactFluorescent Light bulbs (CFLs) at subsidized prices to increaseFluorescent Light bulbs (CFLs) at subsidized prices to increaseenergy efficiency and reduce energy costs for the poor.energy efficiency and reduce energy costs for the poor.

The team wants to know how to deliver the greatest number ofThe team wants to know how to deliver the greatest number of

CFLs under budget constraints. What is the optimal value forCFLs under budget constraints. What is the optimal value forthe subsidy value? What is the best delivery mechanism?the subsidy value? What is the best delivery mechanism?

During the first six months of CFL distribution, the electricDuring the first six months of CFL distribution, the electriccompany will experiment with alterative subsidy values (high,company will experiment with alterative subsidy values (high,medium, low) and distribution mechanisms (local market,medium, low) and distribution mechanisms (local market,

company distribution) in different localities. Localities will becompany distribution) in different localities. Localities will berandomly selected to one or the other treatment.randomly selected to one or the other treatment.

At the end of the first six months, the company with evaluateAt the end of the first six months, the company with evaluatewhich model was most costwhich model was most cost--efficient (use of CFLs per dollarefficient (use of CFLs per dollarspent), and implement that model for the rest of the project.spent), and implement that model for the rest of the project.


15/37

Preparation through appraisalPreparation through appraisal


16/37

Define project developmentDefine project development

objectives and results frameworkobjectives and results framework This activityThis activity

clarifies the results chain for the project,clarifies the results chain for the project,

identifies the outcomes of interest and the indicatorsidentifies the outcomes of interest and the indicators

best suited to measure changes in those outcomes,best suited to measure changes in those outcomes,andand

the expected time horizon for changes in thosethe expected time horizon for changes in those

outcomes.outcomes.

This will provide the lead researcher with theThis will provide the lead researcher with theproject specific variables that must be includedproject specific variables that must be included

in the survey questionnaire and a notion ofin the survey questionnaire and a notion of

timing for scheduling data collection.timing for scheduling data collection.


17/37

Work out project design featuresWork out project design features

that will affect evaluation designthat will affect evaluation design Target population and rules of selectionTarget population and rules of selection

This provides the evaluator with the universe for theThis provides the evaluator with the universe for the

treatment and comparison sampletreatment and comparison sample

Roll out planRoll out plan This provide the evaluation with a framework forThis provide the evaluation with a framework for

timing data collection and, possibly, an opportunity totiming data collection and, possibly, an opportunity to

define a comparison groupdefine a comparison group

The impact evaluator will work iteratively with theThe impact evaluator will work iteratively with theteam to agree on selection and roll out thatteam to agree on selection and roll out that

allows for rigorous evaluation (F&A)allows for rigorous evaluation (F&A)


18/37

Narrow down the questions for theNarrow down the questions for the

evaluationevaluation

Questions aimed at measuring the impact ofQuestions aimed at measuring the impact of

the project on a set of outcomes, andthe project on a set of outcomes, and

Questions aimed at measuring the relativeQuestions aimed at measuring the relative

effectiveness of different features of theeffectiveness of different features of the

projectproject


19/37

What is your hypothesis? (Results framework)What is your hypothesis? (Results framework) By expanding water supply, the use of clean water will increase,By expanding water supply, the use of clean water will increase,

water borne disease decline, and health status improvewater borne disease decline, and health status improve

What is the evaluation question?What is the evaluation question? Does improved water supply result in better health outcomes?Does improved water supply result in better health outcomes?

How can do you test the hypothesis?How can do you test the hypothesis? The government might randomly assign areas for expansion in waterThe government might randomly assign areas for expansion in water

supply during the first and second phase of the programsupply during the first and second phase of the program

What will you measure?What will you measure?

Measure the change in health outcomes in phase I areas relative toMeasure the change in health outcomes in phase I areas relative tothe change in outcomes in phase II areas. Outcomes will include usethe change in outcomes in phase II areas. Outcomes will include useof safe water (Sof safe water (S--T), incidence of diarrhea (S/MT), incidence of diarrhea (S/M--T), and health statusT), and health status(L(L--T, depending on when phase II occurs). Add other outcomes.T, depending on when phase II occurs). Add other outcomes.

What will you do with the results?What will you do with the results? If the hypothesis proves true go to phase II; if false, modify policy.If the hypothesis proves true go to phase II; if false, modify policy.

Questions aimed at measuring the impact ofQuestions aimed at measuring the impact of

the projectthe project are relatively straightforwardare relatively straightforward


20/37

Questions aimed at measuring the relativeQuestions aimed at measuring the relative

effectiveness of different project featureseffectiveness of different project features

require identifying the tough design choices on therequire identifying the tough design choices on thetabletable

What is the issue?What is the issue?

What is the best package of products or services?What is the best package of products or services? Where do you start from?Where do you start from?

What package is the government delivering now?What package is the government delivering now?

Which changes do you or the government think couldWhich changes do you or the government think couldbe made to improve effectiveness?be made to improve effectiveness?


21/37

How do you test it?How do you test it? The government might agree to provide a package to aThe government might agree to provide a package to a

randomly selected group of households and anotherrandomly selected group of households and anotherpackage to another group of households to see how thepackage to another group of households to see how thetwo package performtwo package perform

What will you measure?What will you measure?

The average change in relevant outcomes forThe average change in relevant outcomes forhouseholds receiving one package versus the same forhouseholds receiving one package versus the same forhouseholds receiving the other packagehouseholds receiving the other package

What will you do with the results?What will you do with the results? The package that is most effective in deliveringThe package that is most effective in delivering

desirable outcomes becomes the one adopted by thedesirable outcomes becomes the one adopted by theproject from the evaluation onwardsproject from the evaluation onwards


22/37

Application, features that should beApplication, features that should be

tested early ontested early on

Early testing of project features (say 6 months toEarly testing of project features (say 6 months to1 year) can provide the team with the1 year) can provide the team with theinformation needed to adjust the project early oninformation needed to adjust the project early on

in the direction most likely to deliver success.in the direction most likely to deliver success. Features might include:Features might include:

alternative modes of delivery (e.g. schoolalternative modes of delivery (e.g. school--based vs.based vs.vertical delivery),vertical delivery),

alternative packages of outputs (e.g. awarenessalternative packages of outputs (e.g. awarenesscampaigns vs. legal services), orcampaigns vs. legal services), or

different pricing schemes (e.g. alternative subsidydifferent pricing schemes (e.g. alternative subsidylevels).levels).


23/37


24/37

Develop identification strategyDevelop identification strategy((to identify the impact of the project separately from changes due toto identify the impact of the project separately from changes due to

other causesother causes ))

One the questions are defined, the leadOne the questions are defined, the leadresearcher selects one or more comparisonresearcher selects one or more comparisongroups against which to measure results in thegroups against which to measure results in the

treatment group.treatment group. The rigor with which the comparison group isThe rigor with which the comparison group is

selected will determine the reliability of theselected will determine the reliability of theimpact estimates.impact estimates.

Rigor?Rigor? MoreMore--same observables and unobservablessame observables and unobservables

(experimental),(experimental),

LessLess--same observables (nonsame observables (non--experimental)experimental)


25/37

Explore Existing DataExplore Existing Data

Explore what data exists that might be relevantExplore what data exists that might be relevantfor use in the evaluation.for use in the evaluation. Discuss with the agencies of the national statisticalDiscuss with the agencies of the national statistical

system and universities to identify existing datasystem and universities to identify existing data

sources and future data collection plans.sources and future data collection plans. Record data periodicity, quality, variablesRecord data periodicity, quality, variables

covered and sampling frame and sample size,covered and sampling frame and sample size,forfor CensusesCensuses

Surveys (household, firms, facility, etc)Surveys (household, firms, facility, etc)

Administrative dataAdministrative data

Data from the project monitoring systemData from the project monitoring system


26/37

New DataNew Data Start identifying additional data collection needsStart identifying additional data collection needs..

Data for impact evaluation must be representative of treatment andData for impact evaluation must be representative of treatment andcomparison groupcomparison group

Questionnaires must include outcomes of interest (consumption,Questionnaires must include outcomes of interest (consumption,income, assets etc), questions about the program in question andincome, assets etc), questions about the program in question andquestions about other programsquestions about other programs

The data might be at household, community, firm, facility, or farmThe data might be at household, community, firm, facility, or farmlevels and might be combined with specialty data such as thoselevels and might be combined with specialty data such as thosefrom water or land quality tests.from water or land quality tests.

Investigate synergies with other projects to combine dataInvestigate synergies with other projects to combine datacollection efforts and/or explore existing data collectioncollection efforts and/or explore existing data collectionefforts on which the new data collection could piggy backefforts on which the new data collection could piggy back

Develop a data strategy for the impact evaluation including:Develop a data strategy for the impact evaluation including: The timing for data collectionThe timing for data collection

The variables neededThe variables needed

The sampleThe sample

Plans to integrate data from other sourcesPlans to integrate data from other sources


27/37

Prepare for collecting dataPrepare for collecting data

Identify data collection agencyIdentify data collection agency Lead researcher will work with the dataLead researcher will work with the data

collection agency to design sample, and traincollection agency to design sample, and trainenumeratorsenumerators

Lead researcher will prepare surveyLead researcher will prepare surveyquestionnaire or questionnaire module asquestionnaire or questionnaire module asneededneeded

PrePre--testing survey instrument may take place attesting survey instrument may take place atthis stage to finalize instrumentsthis stage to finalize instruments

If financed with outside funds, baseline can nowIf financed with outside funds, baseline can nowgo to the field. If financed by project funds,go to the field. If financed by project funds,baseline will go to the field just afterbaseline will go to the field just aftereffectiveness but before implementation startseffectiveness but before implementation starts


28/37

Develop a Financial PlanDevelop a Financial Plan

Costs:Costs: Lead researcher and researcher team,Lead researcher and researcher team,

Data collection,Data collection,

Supervision andSupervision and

DisseminationDissemination

Finances:Finances: BB,BB,

Trust fund,Trust fund,

Research grants,Research grants, Project funds, orProject funds, or

Other donor fundsOther donor funds


29/37

HousekeepingHousekeeping Initiate an IE activity. The IE code in SAP is a way ofInitiate an IE activity. The IE code in SAP is a way of

formalizing evaluation activities. The IE code recognizesformalizing evaluation activities. The IE code recognizesthe evaluation as a separate AAA product.the evaluation as a separate AAA product. Prepare concept notePrepare concept note

Identify peer reviewersIdentify peer reviewers impact evaluation and sector specialistimpact evaluation and sector specialist

Carry out review processCarry out review process

Appraisal documentsAppraisal documents Include in the project description plans to modify project overtimeInclude in the project description plans to modify project overtime

to incorporate resultsto incorporate results

Work the impact evaluation into the M&E section of the PAD andWork the impact evaluation into the M&E section of the PAD and

Annex 3Annex 3

Include the impact evaluation in the QualityInclude the impact evaluation in the QualityEnhancement Review (TTL).Enhancement Review (TTL).


30/37

Negotiations to CompletionNegotiations to Completion


31/37

Ensure timely implementationEnsure timely implementation

Ensure timely procurement of evaluationEnsure timely procurement of evaluation

services especially contracting the dataservices especially contracting the data

collection, andcollection, and

Supervise timely implementation of theSupervise timely implementation of the

evaluation includingevaluation including

Data collectionData collection

Data analysisData analysis

Dissemination and feedbackDissemination and feedback


32/37

Data collection agency/firmData collection agency/firm

Data collection agency or firm must haveData collection agency or firm must have

technical knowledge and sufficienttechnical knowledge and sufficient

logistical capacity relative to the scale oflogistical capacity relative to the scale of

data collection requireddata collection required

The same agency or firm should beThe same agency or firm should be

expected to do baseline and follow up dataexpected to do baseline and follow up data

collectioncollection


33/37

Baseline data collection and analysisBaseline data collection and analysis

Baseline data collection should be carriedBaseline data collection should be carriedout before program implementationout before program implementationbegins; optimally even before program isbegins; optimally even before program isannouncedannounced

Analysis of baseline data will provideAnalysis of baseline data will provideprogram management with additionalprogram management with additionalinformation that might help finalizeinformation that might help finalizeprogram designprogram design


34/37

FollowFollow--up data collection andup data collection and

analysisanalysis

The timing of followThe timing of follow--up data collectionup data collectionmust reflect the learning strategy adoptedmust reflect the learning strategy adopted

Early data collection will help modifyingEarly data collection will help modifying

programs mid course to maximize longerprograms mid course to maximize longer--term effectivenessterm effectiveness

Later data collection will confirmLater data collection will confirm

achievement of longerachievement of longer--term outcomes andterm outcomes andjustify continued flows of fiscal resourcesjustify continued flows of fiscal resourcesinto the programinto the program


35/37

DisseminationDissemination

Implement plan for dissemination of evaluation resultsImplement plan for dissemination of evaluation resultsensuring that the timing is aligned with governmentsensuring that the timing is aligned with governments

decision making cycle.decision making cycle.

Ensure that results are used to inform projectEnsure that results are used to inform project

management and that available entry points aremanagement and that available entry points areexploited to provide additional feedback to theexploited to provide additional feedback to the

governmentgovernment

Ensure that wider dissemination takes place only afterEnsure that wider dissemination takes place only after

the client has had a chance to preview and discuss thethe client has had a chance to preview and discuss the

resultsresults

Nurture collaboration with government and localNurture collaboration with government and local

researchers throughout the processresearchers throughout the process


36/37

HousekeepingHousekeeping

Involve local project implementation unitInvolve local project implementation unit

and PIU person responsible for monitoringand PIU person responsible for monitoring

and evaluationand evaluation

Put in place arrangements to procure thePut in place arrangements to procure the

impact evaluation work and fund it on timeimpact evaluation work and fund it on time

Use early results to inform midUse early results to inform mid--termterm

reviewreview

Use later results to inform the ICR, CASUse later results to inform the ICR, CAS

and future operationsand future operations


37/37

Concluding remarksConcluding remarks

Making evaluation work for you requires aMaking evaluation work for you requires achange in the culture of project design andchange in the culture of project design andimplementation, one that maximizes the use ofimplementation, one that maximizes the use of

learning to change course when necessary andlearning to change course when necessary andimprove the chances for successimprove the chances for success

Impact evaluation more than a tool is anImpact evaluation more than a tool is anorganizing analytical framework for doing thisorganizing analytical framework for doing this

Thank youThank you

Impact Eval and Project Cycle

Documents

Transcript of Impact Eval and Project Cycle