Introduction to the CARE Design, Monitoring and Evaluation Standards

Introduction to the Introduction to the CARE InternationalCARE International

Design, Monitoring & Design, Monitoring & Evaluation (DME) Evaluation (DME)

StandardsStandardsby Jim Rugh by Jim Rugh

November, 2001November, 2001

Introduction:Introduction:The CARE Impact Guidelines were developed in 1999 and have been widely accepted.

At the second Impact Evaluation Initiative (IEI) conference (in Wood Norton, UK, August 2001) the DME Standards and Guidelines were updated, based on feedback from the DME Capacity Assessments, and to be compatible with new conceptual lenses such as Rights-Based Approaches, Gender Equity & Diversity, Civil Society, etc.

Introduction:Introduction:

This updated presentation introduces the proposed CI DME Standards and core

guidelines.

It begins with a perspective of where these fit within CARE International (CI).

CI Vision and MissionCI Vision and Mission

We seek a world of hope, tolerance and social justice,

where poverty has been overcome and people live in

dignity and security.

Previous list of CI principles


CI Programming PrinciplesCI Programming Principles

Significant ScopeSignificant ScopeFundamental ChangeFundamental Change

Working with Poor PeopleWorking with Poor PeopleParticipationParticipationReplicabilityReplicability

If we adopted the RBA principles



Affirm peoples right to participate in decisionAffirm peoples right to participate in decision--making making processes that affect their lives. processes that affect their lives. Require identifying and seeking to address the root Require identifying and seeking to address the root causes of poverty and suffering. causes of poverty and suffering. Refuse to tolerate discrimination and inequities that Refuse to tolerate discrimination and inequities that impede peace and development. impede peace and development. Hold all of us accountable for respecting and helping to Hold all of us accountable for respecting and helping to protect and fulfil human rights .protect and fulfil human rights .

CI DME Basic StandardsCI DME Basic Standards

Core DME Guidelines

Array of other detailed guidelines



This presentation focuses on the

Basic DME Standards

THE THE

DME DME

CYCLECYCLE

Holist

ic

Analy

sisFocused

strategy

Cohere

nt

informa

tion

system

Reflective practice

We will now begin going over the CI DME Standards. But before we do, lets look at the introductory paragraphs:

These CARE DME standards apply to all CARE programming (including emergencies, rehabilitation and development) and all forms of interventions (direct service delivery, working with or through partners, and policy advocacy).

These standards, as well as accompanying guidelines, should be used to guide the work of project designers; as a checklist for approval of project proposals; as a tool for periodic project self-appraisal; and as a part of project evaluation. The emphasis should not be only on enforcement but also on the strengthening of capacity to be able to meet these standards for program quality. At the time of initial approval, if a project can not meet one or more standards, allow for explanation of why, and what will be done about it. More than a passed/failed checklist, these call for a description of how well a project meets each standard.

CARE International DME StandardsEach CARE project should:

1. be consistent with the CI Vision & Mission, Programming Principles & Values.

Projects should show how they will contribute, ultimately, towards lasting improvements in human well-being, hope, tolerance, social justice, reduction in poverty, and enhanced dignity and security of people.


2. be clearly linked to CO strategyand/or long term programme goal.

Projects should not be isolated, but clearly embedded in long term multi-project programmes and strategic frameworks that address the underlying conditions and root causes of poverty and social injustice.

Food Food Production Production

Project Project Goal:Goal:

Increase Increase availability availability of food to of food to

householdshouseholds

To achieve impact all of these need to address the same target population.

Program GoalProgram Goal: Reduce : Reduce childhood malnutritionchildhood malnutrition

Health Health Project Project Goal:Goal:

Decrease Decrease diarrheal diarrheal disease disease among among

childrenchildren

Nutrition Nutrition Project Project Goal:Goal:

Improve Improve quality of quality of

food fed to food fed to childrenchildren

Program and project goals at Program and project goals at impact impact level

ASSUMPTION(that others will do this)

CARE project

levelPARTNER will do this


3. Ensure the active participation and influence of stakeholders in its analysis, design, implementation, and M&E processes.Every project should be explicit about its process of participation and consultation, aiming for openness and transparency.

Assessment of Assessment of CommunityCommunityneeds and needs and

assetsassets

CAREs CAREs experience experience

and and comparative comparative advantageadvantage

DONORSDONORSInterestsInterests

PARTNERS PARTNERS interests and interests and capabilitiescapabilities

COMMUNITYCOMMUNITY

CARECARE

DONORSDONORS PARTNERSPARTNERS

VIABLE VIABLE PROJECTPROJECT


4. Have a design that is based on a holistic analysis of the needs and rights of the target population and the underlying causes of their conditions of poverty and social justice. It should also examine the opportunities and risks inherent in the potential interventions.

Elements of Household Livelihood Security

FoodFood

HealthHealth

EducationEducation ShelterShelter Community Community EmpowermentEmpowerment

WaterWater

Means: Income, skills, time

Context: Social, Political and

Cultural

EnvironmentEnvironmentEconomic Economic SecuritySecurity

Whether these are described as needs or

rights, holistic diagnosis implies taking them all

into account.

COMMUNITY

DISTRICT

STATE

NATION

WORLD

CBOs

NGOs

National / sub-regional / local

Government

Bi- / Multi-lateral Agencies

Private Sector

Social Networks

CAREs options include working with or through partners, including policy advocacy Bi- / Multi-lateral

Direct Service

Through whatever Through whatever mode, it needs to be mode, it needs to be shown that ultimately shown that ultimately there will be impact there will be impact on Household on Household Livelihood Security.Livelihood Security.

Social Networks

CBOs

NGOs

Agencies

National / sub-regional / local

Government

Private Sector

GE

R

GE

R

UM

GHTS

UM

GHTS

Other lenses may be used as well

H

NDE

NDE

AN RI

HCIVIL SOCIETYAN RI


5. Use a logical framework that explains how the project will contribute to an ultimate impact upon the lives of members of a defined target population.Specify level of intervention (household, community, institutional, societal); identify key assumptions and provide validation for its central hypothesis.

Converting the Converting the diagnostic diagnostic

assessment of assessment of needs/needs/opportunitiesopportunities

into a logical into a logical problem/problem/solution solution tree tree is the essential step is the essential step

in project designin project design

Consequences Consequences Consequences

PROBLEMPROBLEM

PRIMARY PRIMARY CAUSE 2CAUSE 2

PRIMARYPRIMARYCAUSE 1CAUSE 1

PRIMARY PRIMARY CAUSE 3CAUSE 3

Secondary Secondary cause cause 2.22.2



Tertiary Tertiary cause cause 2.2.12.2.1



High infant mortality rate

Children are malnourishedChildren are malnourished

Diarrheal Diarrheal diseasedisease

Insufficient Insufficient foodfood

Poor quality Poor quality of foodof food

Unsanitary Unsanitary practicespractices

Need for Need for improved health improved health

policiespolicies

Need for strengthened Need for strengthened capacity of health capacity of health

institutionsinstitutions

Flies and Flies and rodentsrodents

Do not use Do not use facilities facilities correctlycorrectly

People do not People do not wash hands wash hands before eatingbefore eating

Lower infant mortality rate

More Children are well nourishedMore Children are well nourished

Less Less diarrhealdiarrhealdiseasedisease

Sufficient Sufficient foodfood

Good quality Good quality of foodof food

Sanitary Sanitary practicespractices

Improved health Improved health policiespolicies

Strengthened capacity Strengthened capacity of health institutionsof health institutions

Fewer flies Fewer flies and rodentsand rodents

facilities facilities used used

correctlycorrectly

People wash People wash hands before hands before

eatingeating

Common terminology for project hierarchy

OUTCOMESOUTCOMES

OUTPUTS

INPUTS

PROCESSPROCESS

INPUTS Resources needed by project (e.g. funds, staff, commodities)

ACTIVITIES Interventions / processes implemented by project

OUTPUTS Products of project activities

O

U

T

C

O

M

E

S

EFFECTSEFFECTS Changes in individual behavior or systemic competence.

IMPACTIMPACT Equitable and durable improvements in human well-being and social justice

-

-

-

R

E

S

U

L

T

S

-

-

-

CARECAREs terminology for project hierarchyP

R

O

C

E

S

S

Example for childhood malnutrition project

PROGRAM IMPACT PROGRAM IMPACT Reduction in malnutrition rate among children under five

PROJECT IMPACTPROJECT IMPACT Decrease in the incidence of diarrhea

EFFECT Parents of children practice sanitary behaviors

OUTPUT Parents acquire knowledge about sanitary practices

ACTIVITIES Parents given training in sanitary practices

INPUTS Funding obtained, staff trained, training center set up

If sufficient INPUTS are received , then we will be able to do ...

ACTIVITIES (interventions) which should lead to ...

OUTPUTS, which, if our assumptions hold true, the effectiveness can be measured by...

EFFECTSEFFECTS which, if our hypothesis is valid, should be shown to lead to ...

IMPACT !IMPACT !

The quality of each level is measured by the next higher level.

Comparison of logical framework terms used by different agencies

This has been called the Rosetta Stone of LogFrames


6. Set a significant, yet achievableand measurable final goal.

The final goal should be practical and do-able, yet be at the outcome level (intermediary impact or at least effect).A project final goal should clearly contribute to higher level program impact: equitable and durable improvements in human well-being and social justice.

Long-term (ultimate) impact (Strategic Plan or Program level)

Project Effects

SignificantSignificant

PROJECT FINAL GOALPROJECT FINAL GOALShort-term (intermediate) impact

(Achievable & measurable during LOP)

Project Outputs AchievableAchievable


7. be technically, environmentally and socially appropriate. Interventions should be based upon best current practice and on an understanding of the social context and the needs, rights and responsibilities of the stakeholders.


8. Indicate the appropriateness of project costs in light of the selected project strategies and expected outputs and outcomes.Project designers must be able to defend the budget relative to its outputs, scale and anticipated impact. M&E plan should include methods for measuring cost effectiveness.


9. Develop and implement an M&E planand system based on the logical framework that ensures the collection of baseline, monitoring, and final evaluation data, and anticipates how the information will be used for decision making; with a budget that includes adequate amounts for implementing the M&E plan.

M&E Methodologies

Project M&E plans should provide sufficient detail to clearly identify:8evaluation design (based on key questions)8sources of data (evidence to be collected)8means of measurement (how to collect evidence)8schedule for measurement (when)8data processing and analysis8dissemination of information to and utilization

by key stakeholders, and8 responsibilities for each of these processes.

What should be What should be included in budgets included in budgets

to assure to assure accountability?accountability?

ACCOUNTABILITYConsider how many Consider how many

resources we put resources we put into accounting for into accounting for

how funds have been how funds have been spent.

Then consider how Then consider how many resources we many resources we put into monitoring put into monitoring

and evaluating and evaluating results achieved by results achieved by

our projects.spent.

our projects.

FUNDS FUNDS ACCOUNTINGACCOUNTING

M&E of M&E of RESULTSRESULTS


10. Establish a baseline for measuring change in indicators of impact and effect, by conducting a study or survey prior to implementation of project activities.Distinguish between a diagnostic assessment and baseline. The latter focuses on measuring effect & impact indicators required for before-and-after comparison with evaluation.

We need to be We need to be aware of the aware of the

relative focus and relative focus and purposes of purposes of

different forms of different forms of assessmentassessment

Long-Range Strategic Plan

Household Livelihood Security Assessment

Including review of secondary data

More focused


11. Use indicators that are relevant, measurable, verifiable and reliable.Indicators should be capable of yielding data that can reveal vulnerabilities such as gender, age and social class.Both qualitative and quantitative measures are acceptable as long as they can illustrate discernible and significant change.

Indicators

8Indicators should be relevant to the goals they represent, quantitatively or qualitatively measurable, objectively verifiable, reliable, meet international professional standards ...8and yet be understandable and

appreciated by project participants and other stakeholders.

Indicators

Be clear on appropriate levels of indicators:8Indicators that represent different

levels of goals and objectives in the logframe hierarchy8Different levels of precision from

broad issues to objectively verifiable indicators and specific raw data variables to be collected.

Example of levels of precision of indicators

Broad indicator:8 Improve the well-being of childrenObjectively verifiable indicator:8Level of childhood malnutrition measured as %

of children


12. Employ a balance of evaluation methodologies,assure an appropriate level of rigor, and adhere to recognized ethical standards.The minimum is that there should be at least a final evaluation that summarizes the achievements and lessons learned by the project. Evaluation events should utilize a balance of methodological approaches to assure triangulation, and a richness of data.

Appropriate mix of Methods

Employ appropriate combinations of methodologies during assessments, monitoring and evaluation, including:4Qualitative / quantitative4Participatory / extractive4Multi-disciplinary in focus4Secondary / primary data sources

QUANTITATIVEQUANTITATIVE

QUALITATIVEQUALITATIVE

Your numbers Your numbers look impressive, look impressive,

but let me tell you but let me tell you the human the human

interest story.interest story.

Your human interest story

sounds nice, but let me show you the statistics.

DIFFERENT PARADIGMSWe need to be careful that we not get too

locked into the scientific, hypothetico-deductive paradigm (with a clear cause-effect chain attributable to a projects interventions). It is not the only one that can reveal reality in the world where CARE works.

Also, we need to beware of reductionism, especially the attempt to reduce complex realities to superficial, numerical scales -- to quantify the qualitative.

DIFFERENT PARADIGMS

There are other, more culturally-sensitive paradigms, including the perspectives of participant communities themselves, on whether or not they feel their lives have improved.

We have to ask what evidence donors (and the rest of us) would accept that impact has been achieved.

Participatory approaches should be used as much as possible

But even they should be used with appropriate rigor: how many peoples perspectives

contributed to the story?


13. be informed by and contribute to ongoing learning within and outside CARE.Relevant research and previous evaluations should inform the project design. Also, lessons learned from a project should be adequately documented for utilization in the design of other projects. Learning should be an organization-wide priority, supported by frequent meta-evaluations.

Thats the end of the introduction to the CI DME Standards

If youve had enough you can end this presentation here.

Or proceed to learn more about basic DME guidelines.

A little more on basic DME methodologies

consistent with the DME Standards

An Overview of the DME Cycle in CARE

1) Long-Range Strategic Plan2) Diagnosis, including Participatory Assessment of

Community Needs and Opportunities3) Project Design (summarized in logical framework) Submission of proposal to donors

4) Detailed implementation plan, including detailed M&E plan based on life-of-project evaluation design

5) Baseline study6) Monitoring of implementation7) Evaluations: Mid-term + Final (+ Post-project)8) Apply lessons learned to next planning cycle

Illustrating the need for quasiIllustrating the need for quasi--experimental experimental longitudinal time series evaluation design

An introduction to various evaluation designsAn introduction to various evaluation designs

longitudinal time series evaluation design

Project participantsProject participants

Comparison groupComparison group

end of project end of project evaluation

post project post project evaluation

baselinebaselineevaluation evaluation

scale of major impact indicatorscale of major impact indicator

Suggested criteria for determining appropriate design for a projects

evaluation plan

For what reasons should longitudinal, quasi-experimental research/evaluation designs be used? To test hypothesis (i.e... correlation between interventions

outputs and impact) which has not been previously proven through other evaluations or research

Pilot project to serve as model for wide multiplication When donor insists that it is necessary to prove impact and

attribution Donor willing to dedicate significant amount of budget (e.g.

10%-25% or even more) for research (M&E)

Suggested criteria for determining appropriate design for a projects

evaluation plan When might quasi-experimental impact

evaluation design notnot be necessary? Proven intervention, correlation between interventions and

impact previously proven in acceptably similar conditions Only need to verify that implementation and outputs comply

with standards Reliable, valid and relevant secondary data available When it would be unreasonable (or unethical) to use a

control group Short-term (i.e. emergency); Questionable security situation Only 3%-10% budget available for M&E.

Different levels of rigorDifferent levels of rigordepends on source of evidence; level of confidence; use of information

Level 0: Decision-makers impressions based on anecdotes and sound bytes heard during brief encounters (hallway gossip), mostly intuition; Level of confidence +/- 50%; Decision made in a few seconds

Level 1: A few people are asked their perspectives about project;P= +/- 40% Decision made in a few minutes

Level 3: A rapid survey is conducted on a convenient sample of participants; P= +/- 15% Decision maker reads 10-page summary of report

Level 2: A fairly good mix of people are asked their perspectives about project; P= +/- 25% Decision maker reads at least executive summary of report

Level 4: Good sampling and data collection methods used to gather data which is representative of target population; P= +/- 5% Decision maker reads full report

Level 5: A thorough research project is undertaken to conduct in-depth analysis of situation; P= +/- 1% Book published!

Objective,Objective, High precision, More time & expenseHigh precision, More time & expense

Subjective , Sloppy, Quick & cheapSubjective , Sloppy, Quick & cheap

Determining appropriate Determining appropriate precision precision and mix of and mix of multiple methodsmultiple methods

High rigor, high quality, more time & expenseHigh rigor, high quality, more time & expense

P

a

r

t

i

c

i

p

a

t

o

r

y

-

-

-

Q

u

a

l

i

t

a

t

i

v

e

KeyKeyInformantInformantinterviewsinterviews

FocusFocusGroupsGroups

HHHHsurveyssurveys

NutritionalNutritionalmeasurementsmeasurements

LargeLargegroupgroup

FocusFocusGroupsGroupsHHHH

surveyssurveys

NutritionalNutritionalmeasurementsmeasurements

E

x

t

r

a

c

t

i

v

e

-

-

-

Q

u

a

n

t

i

t

a

t

i

v

e

Low rigor, questionable quality, quick and cheapLow rigor, questionable quality, quick and cheap

Determining appropriate Determining appropriate levels of precision levels of precision for for

events in a projects evaluation planevents in a projects evaluation plan

AnnualAnnualselfself--evaluationevaluation

MidMid--termtermevaluationevaluation

BaselineBaselinestudystudy

NeedsNeedsassessmentassessment

FinalFinalevaluationevaluation

Time during project life cycleTime during project life cycle

Special Special StudyStudy

Same level of rigorSame level of rigor

High rigorHigh rigor

Low rigorLow rigor2

3

4

Project conceptProject conceptProject Design Logframe M&E plan

INDICATORSINDICATORS INDICATORS

Outputs Activities Inputs

Effects Impact

INDICATORS INDICATORS

The The problemproblemTarget groupTarget group

ASSUMPTIONS / EXTERNAL FACTORS

INDICATORS

Project hypothesis: outputs + valid assumptions will Project hypothesis: outputs + valid assumptions will lead to outcomeslead to outcomes

Diagnosis

EvaluationOf Outcome / Purpose

Upper Logframe

Monitoring Of Process / Operation

Lower Logframe

A good M&E plan will show all of these dimensions and how they relate

The Project Implementation Process

Effects Impact Outputs Activities

Project Efficiency

Project Effectiveness

Was our hypothesis valid?Was our hypothesis valid?Are we doing well?Are we doing well?

Inputs

Did we do the right thing in a worthwhile way?Did we do the right thing in a worthwhile way?

Three Major Components of Good DME

L

o

g

i

c

a

l

F

r

a

m

e

w

o

r

k

D

e

t

a

i

l

e

d

M

&

E

P

l

a

n

S

y

s

t

e

m

a

t

i

c

D

M

E

-

I

S

Levels of DME-IS (from recent DME-IS conference)

Level 1: paper-based, manual system

1

Information Requirements

L

e

v

e

l

o

f

s

o

p

h

i

s

t

i

c

a

t

i

o

n

23

44

Level 2: Common software programs, e.g. Excel, Word

Level 3: Database & analysis software

Level 4: Fully integratedsoftware package

We need to recognize that not all We need to recognize that not all projects are ready for level 4 projects are ready for level 4

DMEDME--IS. Even at level 1 IS. Even at level 1 (manual systems) there is room (manual systems) there is room

for improved systemization. for improved systemization.

In fact, systemize before you In fact, systemize before you automate needs to be the guiding automate needs to be the guiding

principle behind all DMEprinciple behind all DME--IS.IS.

The quality of DME in a project The quality of DME in a project can be greatly enhanced by using can be greatly enhanced by using

a fully integrated DMEa fully integrated DME--IS (like IS (like MER)MER), for it depends upon, , for it depends upon, complements, and makes complements, and makes

functional a logical logframe and functional a logical logframe and well detailed M&E plan. Not only well detailed M&E plan. Not only

for one project, but related for one project, but related projects within a larger program.projects within a larger program.

Over-all Principles of good DME

8Holistic diagnosis of needs and opportunities which includes community participation8Logical project designs8Systematized monitoring systems8Quality evaluations that measure impact, are credible and useful

8Decision makers think evaluatively

Whats meant by thinking evaluatively?4Reality checks: We all need to seek

objective feedback, gain perspectives on our work; learn lessons and apply them.4Rational decision-making: Before

making decisions be clear on what evidence we have, from whom it was obtained, and how reliable it is.

Whats meant by thinking evaluatively?4Challenge paradigms: -- Our own as

well as others. Be visionary. Think out of the box. Ask what other perspectives would be informative.

Whats meant by thinking evaluatively?4Be accountable: We owe it to our clients

(intended beneficiaries as well as donors) to document not only our use of inputs and production of outputs, but also what outcomes are achieved; i.e. what difference our projects have made in the lives of real people.

In these ways we can assure that In these ways we can assure that

all of us working togetherall of us working togetherwill help make the world a will help make the world a

better place for allbetter place for all !!!!

Nagarkot, Nepal; photo by Richard Caldwell

Introduction to the CARE InternationalDesign, Monitoring & Evaluation (DME) Standardsby Jim Rugh November, 2001Example for childhood malnutrition projectThe quality of each level is measured by the next higher level.ACCOUNTABILITYDIFFERENT PARADIGMSDIFFERENT PARADIGMSA little more on basic DME methodologies consistent with the DME StandardsAn Overview of the DME Cycle in CARESuggested criteria for determining appropriate design for a projects evaluation planSuggested criteria for determining appropriate design for a projects evaluation planOver-all Principles of good DMEWhats meant by thinking evaluatively?Whats meant by thinking evaluatively?Whats meant by thinking evaluatively?In these ways we can assure that

Introduction to the CARE Design, Monitoring and Evaluation Standards

Documents

Transcript of Introduction to the CARE Design, Monitoring and Evaluation Standards