Introduction to the Introduction to the CARE InternationalCARE International
Design, Monitoring & Design, Monitoring & Evaluation (DME) Evaluation (DME)
StandardsStandardsby Jim Rugh by Jim Rugh
November, 2001November, 2001
Introduction:Introduction:The CARE Impact Guidelines were developed in 1999 and have been widely accepted.
At the second Impact Evaluation Initiative (IEI) conference (in Wood Norton, UK, August 2001) the DME Standards and Guidelines were updated, based on feedback from the DME Capacity Assessments, and to be compatible with new conceptual lenses such as Rights-Based Approaches, Gender Equity & Diversity, Civil Society, etc.
Introduction:Introduction:
This updated presentation introduces the proposed CI DME Standards and core
guidelines.
It begins with a perspective of where these fit within CARE International (CI).
CI Vision and MissionCI Vision and Mission
We seek a world of hope, tolerance and social justice,
where poverty has been overcome and people live in
dignity and security.
Previous list of CI principles
CI Vision and MissionCI Vision and Mission
CI Programming PrinciplesCI Programming Principles
Significant ScopeSignificant ScopeFundamental ChangeFundamental Change
Working with Poor PeopleWorking with Poor PeopleParticipationParticipationReplicabilityReplicability
If we adopted the RBA principles
CI Vision and MissionCI Vision and Mission
CI Programming PrinciplesCI Programming Principles
Affirm peoples right to participate in decisionAffirm peoples right to participate in decision--making making processes that affect their lives. processes that affect their lives. Require identifying and seeking to address the root Require identifying and seeking to address the root causes of poverty and suffering. causes of poverty and suffering. Refuse to tolerate discrimination and inequities that Refuse to tolerate discrimination and inequities that impede peace and development. impede peace and development. Hold all of us accountable for respecting and helping to Hold all of us accountable for respecting and helping to protect and fulfil human rights .protect and fulfil human rights .
CI DME Basic StandardsCI DME Basic Standards
Core DME Guidelines
Array of other detailed guidelines
CI Vision and MissionCI Vision and Mission
CI Programming PrinciplesCI Programming Principles
This presentation focuses on the
Basic DME Standards
THE THE
DME DME
CYCLECYCLE
Holist
ic
Analy
sisFocused
strategy
Cohere
nt
informa
tion
system
Reflective practice
We will now begin going over the CI DME Standards. But before we do, lets look at the introductory paragraphs:
These CARE DME standards apply to all CARE programming (including emergencies, rehabilitation and development) and all forms of interventions (direct service delivery, working with or through partners, and policy advocacy).
These standards, as well as accompanying guidelines, should be used to guide the work of project designers; as a checklist for approval of project proposals; as a tool for periodic project self-appraisal; and as a part of project evaluation. The emphasis should not be only on enforcement but also on the strengthening of capacity to be able to meet these standards for program quality. At the time of initial approval, if a project can not meet one or more standards, allow for explanation of why, and what will be done about it. More than a passed/failed checklist, these call for a description of how well a project meets each standard.
CARE International DME StandardsEach CARE project should:
1. be consistent with the CI Vision & Mission, Programming Principles & Values.
Projects should show how they will contribute, ultimately, towards lasting improvements in human well-being, hope, tolerance, social justice, reduction in poverty, and enhanced dignity and security of people.
CARE International DME StandardsEach CARE project should:
2. be clearly linked to CO strategyand/or long term programme goal.
Projects should not be isolated, but clearly embedded in long term multi-project programmes and strategic frameworks that address the underlying conditions and root causes of poverty and social injustice.
Food Food Production Production
Project Project Goal:Goal:
Increase Increase availability availability of food to of food to
householdshouseholds
To achieve impact all of these need to address the same target population.
Program GoalProgram Goal: Reduce : Reduce childhood malnutritionchildhood malnutrition
Health Health Project Project Goal:Goal:
Decrease Decrease diarrheal diarrheal disease disease among among
childrenchildren
Nutrition Nutrition Project Project Goal:Goal:
Improve Improve quality of quality of
food fed to food fed to childrenchildren
Program and project goals at Program and project goals at impact impact level
ASSUMPTION(that others will do this)
CARE project
levelPARTNER will do this
CARE International DME StandardsEach CARE project should:
3. Ensure the active participation and influence of stakeholders in its analysis, design, implementation, and M&E processes.Every project should be explicit about its process of participation and consultation, aiming for openness and transparency.
Assessment of Assessment of CommunityCommunityneeds and needs and
assetsassets
CAREs CAREs experience experience
and and comparative comparative advantageadvantage
DONORSDONORSInterestsInterests
PARTNERS PARTNERS interests and interests and capabilitiescapabilities
COMMUNITYCOMMUNITY
CARECARE
DONORSDONORS PARTNERSPARTNERS
VIABLE VIABLE PROJECTPROJECT
CARE International DME StandardsEach CARE project should:
4. Have a design that is based on a holistic analysis of the needs and rights of the target population and the underlying causes of their conditions of poverty and social justice. It should also examine the opportunities and risks inherent in the potential interventions.
Elements of Household Livelihood Security
FoodFood
HealthHealth
EducationEducation ShelterShelter Community Community EmpowermentEmpowerment
WaterWater
Means: Income, skills, time
Context: Social, Political and
Cultural
EnvironmentEnvironmentEconomic Economic SecuritySecurity
Whether these are described as needs or
rights, holistic diagnosis implies taking them all
into account.
COMMUNITY
DISTRICT
STATE
NATION
WORLD
CBOs
NGOs
National / sub-regional / local
Government
Bi- / Multi-lateral Agencies
Private Sector
Social Networks
CAREs options include working with or through partners, including policy advocacy Bi- / Multi-lateral
Direct Service
Through whatever Through whatever mode, it needs to be mode, it needs to be shown that ultimately shown that ultimately there will be impact there will be impact on Household on Household Livelihood Security.Livelihood Security.
Social Networks
CBOs
NGOs
Agencies
National / sub-regional / local
Government
Private Sector
GE
R
GE
R
UM
GHTS
UM
GHTS
Other lenses may be used as well
H
NDE
NDE
AN RI
HCIVIL SOCIETYAN RI
CARE International DME StandardsEach CARE project should:
5. Use a logical framework that explains how the project will contribute to an ultimate impact upon the lives of members of a defined target population.Specify level of intervention (household, community, institutional, societal); identify key assumptions and provide validation for its central hypothesis.
Converting the Converting the diagnostic diagnostic
assessment of assessment of needs/needs/opportunitiesopportunities
into a logical into a logical problem/problem/solution solution tree tree is the essential step is the essential step
in project designin project design
Consequences Consequences Consequences
PROBLEMPROBLEM
PRIMARY PRIMARY CAUSE 2CAUSE 2
PRIMARYPRIMARYCAUSE 1CAUSE 1
PRIMARY PRIMARY CAUSE 3CAUSE 3
Secondary Secondary cause cause 2.22.2
Secondary Secondary cause cause 2.32.3
Secondary Secondary cause cause 2.12.1
Tertiary Tertiary cause cause 2.2.12.2.1
Tertiary Tertiary cause cause 2.2.22.2.2
Tertiary Tertiary cause cause 2.2.32.2.3
High infant mortality rate
Children are malnourishedChildren are malnourished
Diarrheal Diarrheal diseasedisease
Insufficient Insufficient foodfood
Poor quality Poor quality of foodof food
Unsanitary Unsanitary practicespractices
Need for Need for improved health improved health
policiespolicies
Need for strengthened Need for strengthened capacity of health capacity of health
institutionsinstitutions
Flies and Flies and rodentsrodents
Do not use Do not use facilities facilities correctlycorrectly
People do not People do not wash hands wash hands before eatingbefore eating
Lower infant mortality rate
More Children are well nourishedMore Children are well nourished
Less Less diarrhealdiarrhealdiseasedisease
Sufficient Sufficient foodfood
Good quality Good quality of foodof food
Sanitary Sanitary practicespractices
Improved health Improved health policiespolicies
Strengthened capacity Strengthened capacity of health institutionsof health institutions
Fewer flies Fewer flies and rodentsand rodents
facilities facilities used used
correctlycorrectly
People wash People wash hands before hands before
eatingeating
Common terminology for project hierarchy
OUTCOMESOUTCOMES
OUTPUTS
INPUTS
PROCESSPROCESS
INPUTS Resources needed by project (e.g. funds, staff, commodities)
ACTIVITIES Interventions / processes implemented by project
OUTPUTS Products of project activities
O
U
T
C
O
M
E
S
EFFECTSEFFECTS Changes in individual behavior or systemic competence.
IMPACTIMPACT Equitable and durable improvements in human well-being and social justice
-
-
-
R
E
S
U
L
T
S
-
-
-
CARECAREs terminology for project hierarchyP
R
O
C
E
S
S
Example for childhood malnutrition project
PROGRAM IMPACT PROGRAM IMPACT Reduction in malnutrition rate among children under five
PROJECT IMPACTPROJECT IMPACT Decrease in the incidence of diarrhea
EFFECT Parents of children practice sanitary behaviors
OUTPUT Parents acquire knowledge about sanitary practices
ACTIVITIES Parents given training in sanitary practices
INPUTS Funding obtained, staff trained, training center set up
If sufficient INPUTS are received , then we will be able to do ...
ACTIVITIES (interventions) which should lead to ...
OUTPUTS, which, if our assumptions hold true, the effectiveness can be measured by...
EFFECTSEFFECTS which, if our hypothesis is valid, should be shown to lead to ...
IMPACT !IMPACT !
The quality of each level is measured by the next higher level.
Comparison of logical framework terms used by different agencies
This has been called the Rosetta Stone of LogFrames
CARE International DME StandardsEach CARE project should:
6. Set a significant, yet achievableand measurable final goal.
The final goal should be practical and do-able, yet be at the outcome level (intermediary impact or at least effect).A project final goal should clearly contribute to higher level program impact: equitable and durable improvements in human well-being and social justice.
Long-term (ultimate) impact (Strategic Plan or Program level)
Project Effects
SignificantSignificant
PROJECT FINAL GOALPROJECT FINAL GOALShort-term (intermediate) impact
(Achievable & measurable during LOP)
Project Outputs AchievableAchievable
CARE International DME StandardsEach CARE project should:
7. be technically, environmentally and socially appropriate. Interventions should be based upon best current practice and on an understanding of the social context and the needs, rights and responsibilities of the stakeholders.
CARE International DME StandardsEach CARE project should:
8. Indicate the appropriateness of project costs in light of the selected project strategies and expected outputs and outcomes.Project designers must be able to defend the budget relative to its outputs, scale and anticipated impact. M&E plan should include methods for measuring cost effectiveness.
CARE International DME StandardsEach CARE project should:
9. Develop and implement an M&E planand system based on the logical framework that ensures the collection of baseline, monitoring, and final evaluation data, and anticipates how the information will be used for decision making; with a budget that includes adequate amounts for implementing the M&E plan.
M&E Methodologies
Project M&E plans should provide sufficient detail to clearly identify:8evaluation design (based on key questions)8sources of data (evidence to be collected)8means of measurement (how to collect evidence)8schedule for measurement (when)8data processing and analysis8dissemination of information to and utilization
by key stakeholders, and8 responsibilities for each of these processes.
What should be What should be included in budgets included in budgets
to assure to assure accountability?accountability?
ACCOUNTABILITYConsider how many Consider how many
resources we put resources we put into accounting for into accounting for
how funds have been how funds have been spent.
Then consider how Then consider how many resources we many resources we put into monitoring put into monitoring
and evaluating and evaluating results achieved by results achieved by
our projects.spent.
our projects.
FUNDS FUNDS ACCOUNTINGACCOUNTING
M&E of M&E of RESULTSRESULTS
CARE International DME StandardsEach CARE project should:
10. Establish a baseline for measuring change in indicators of impact and effect, by conducting a study or survey prior to implementation of project activities.Distinguish between a diagnostic assessment and baseline. The latter focuses on measuring effect & impact indicators required for before-and-after comparison with evaluation.
We need to be We need to be aware of the aware of the
relative focus and relative focus and purposes of purposes of
different forms of different forms of assessmentassessment
Long-Range Strategic Plan
Household Livelihood Security Assessment
Including review of secondary data
More focused
CARE International DME StandardsEach CARE project should:
11. Use indicators that are relevant, measurable, verifiable and reliable.Indicators should be capable of yielding data that can reveal vulnerabilities such as gender, age and social class.Both qualitative and quantitative measures are acceptable as long as they can illustrate discernible and significant change.
Indicators
8Indicators should be relevant to the goals they represent, quantitatively or qualitatively measurable, objectively verifiable, reliable, meet international professional standards ...8and yet be understandable and
appreciated by project participants and other stakeholders.
Indicators
Be clear on appropriate levels of indicators:8Indicators that represent different
levels of goals and objectives in the logframe hierarchy8Different levels of precision from
broad issues to objectively verifiable indicators and specific raw data variables to be collected.
Example of levels of precision of indicators
Broad indicator:8 Improve the well-being of childrenObjectively verifiable indicator:8Level of childhood malnutrition measured as %
of children
CARE International DME StandardsEach CARE project should:
12. Employ a balance of evaluation methodologies,assure an appropriate level of rigor, and adhere to recognized ethical standards.The minimum is that there should be at least a final evaluation that summarizes the achievements and lessons learned by the project. Evaluation events should utilize a balance of methodological approaches to assure triangulation, and a richness of data.
Appropriate mix of Methods
Employ appropriate combinations of methodologies during assessments, monitoring and evaluation, including:4Qualitative / quantitative4Participatory / extractive4Multi-disciplinary in focus4Secondary / primary data sources
QUANTITATIVEQUANTITATIVE
QUALITATIVEQUALITATIVE
Your numbers Your numbers look impressive, look impressive,
but let me tell you but let me tell you the human the human
interest story.interest story.
Your human interest story
sounds nice, but let me show you the statistics.
DIFFERENT PARADIGMSWe need to be careful that we not get too
locked into the scientific, hypothetico-deductive paradigm (with a clear cause-effect chain attributable to a projects interventions). It is not the only one that can reveal reality in the world where CARE works.
Also, we need to beware of reductionism, especially the attempt to reduce complex realities to superficial, numerical scales -- to quantify the qualitative.
DIFFERENT PARADIGMS
There are other, more culturally-sensitive paradigms, including the perspectives of participant communities themselves, on whether or not they feel their lives have improved.
We have to ask what evidence donors (and the rest of us) would accept that impact has been achieved.
Participatory approaches should be used as much as possible
But even they should be used with appropriate rigor: how many peoples perspectives
contributed to the story?
CARE International DME StandardsEach CARE project should:
13. be informed by and contribute to ongoing learning within and outside CARE.Relevant research and previous evaluations should inform the project design. Also, lessons learned from a project should be adequately documented for utilization in the design of other projects. Learning should be an organization-wide priority, supported by frequent meta-evaluations.
Thats the end of the introduction to the CI DME Standards
If youve had enough you can end this presentation here.
Or proceed to learn more about basic DME guidelines.
A little more on basic DME methodologies
consistent with the DME Standards
An Overview of the DME Cycle in CARE
1) Long-Range Strategic Plan2) Diagnosis, including Participatory Assessment of
Community Needs and Opportunities3) Project Design (summarized in logical framework) Submission of proposal to donors
4) Detailed implementation plan, including detailed M&E plan based on life-of-project evaluation design
5) Baseline study6) Monitoring of implementation7) Evaluations: Mid-term + Final (+ Post-project)8) Apply lessons learned to next planning cycle
Illustrating the need for quasiIllustrating the need for quasi--experimental experimental longitudinal time series evaluation design
An introduction to various evaluation designsAn introduction to various evaluation designs
longitudinal time series evaluation design
Project participantsProject participants
Comparison groupComparison group
end of project end of project evaluation
post project post project evaluation
baselinebaselineevaluation evaluation
scale of major impact indicatorscale of major impact indicator
Suggested criteria for determining appropriate design for a projects
evaluation plan
For what reasons should longitudinal, quasi-experimental research/evaluation designs be used? To test hypothesis (i.e... correlation between interventions
outputs and impact) which has not been previously proven through other evaluations or research
Pilot project to serve as model for wide multiplication When donor insists that it is necessary to prove impact and
attribution Donor willing to dedicate significant amount of budget (e.g.
10%-25% or even more) for research (M&E)
Suggested criteria for determining appropriate design for a projects
evaluation plan When might quasi-experimental impact
evaluation design notnot be necessary? Proven intervention, correlation between interventions and
impact previously proven in acceptably similar conditions Only need to verify that implementation and outputs comply
with standards Reliable, valid and relevant secondary data available When it would be unreasonable (or unethical) to use a
control group Short-term (i.e. emergency); Questionable security situation Only 3%-10% budget available for M&E.
Different levels of rigorDifferent levels of rigordepends on source of evidence; level of confidence; use of information
Level 0: Decision-makers impressions based on anecdotes and sound bytes heard during brief encounters (hallway gossip), mostly intuition; Level of confidence +/- 50%; Decision made in a few seconds
Level 1: A few people are asked their perspectives about project;P= +/- 40% Decision made in a few minutes
Level 3: A rapid survey is conducted on a convenient sample of participants; P= +/- 15% Decision maker reads 10-page summary of report
Level 2: A fairly good mix of people are asked their perspectives about project; P= +/- 25% Decision maker reads at least executive summary of report
Level 4: Good sampling and data collection methods used to gather data which is representative of target population; P= +/- 5% Decision maker reads full report
Level 5: A thorough research project is undertaken to conduct in-depth analysis of situation; P= +/- 1% Book published!
Objective,Objective, High precision, More time & expenseHigh precision, More time & expense
Subjective , Sloppy, Quick & cheapSubjective , Sloppy, Quick & cheap
Determining appropriate Determining appropriate precision precision and mix of and mix of multiple methodsmultiple methods
High rigor, high quality, more time & expenseHigh rigor, high quality, more time & expense
P
a
r
t
i
c
i
p
a
t
o
r
y
-
-
-
Q
u
a
l
i
t
a
t
i
v
e
KeyKeyInformantInformantinterviewsinterviews
FocusFocusGroupsGroups
HHHHsurveyssurveys
NutritionalNutritionalmeasurementsmeasurements
LargeLargegroupgroup
FocusFocusGroupsGroupsHHHH
surveyssurveys
NutritionalNutritionalmeasurementsmeasurements
E
x
t
r
a
c
t
i
v
e
-
-
-
Q
u
a
n
t
i
t
a
t
i
v
e
Low rigor, questionable quality, quick and cheapLow rigor, questionable quality, quick and cheap
Determining appropriate Determining appropriate levels of precision levels of precision for for
events in a projects evaluation planevents in a projects evaluation plan
AnnualAnnualselfself--evaluationevaluation
MidMid--termtermevaluationevaluation
BaselineBaselinestudystudy
NeedsNeedsassessmentassessment
FinalFinalevaluationevaluation
Time during project life cycleTime during project life cycle
Special Special StudyStudy
Same level of rigorSame level of rigor
High rigorHigh rigor
Low rigorLow rigor2
3
4
Project conceptProject conceptProject Design Logframe M&E plan
INDICATORSINDICATORS INDICATORS
Outputs Activities Inputs
Effects Impact
INDICATORS INDICATORS
The The problemproblemTarget groupTarget group
ASSUMPTIONS / EXTERNAL FACTORS
INDICATORS
Project hypothesis: outputs + valid assumptions will Project hypothesis: outputs + valid assumptions will lead to outcomeslead to outcomes
Diagnosis
EvaluationOf Outcome / Purpose
Upper Logframe
Monitoring Of Process / Operation
Lower Logframe
A good M&E plan will show all of these dimensions and how they relate
The Project Implementation Process
Effects Impact Outputs Activities
Project Efficiency
Project Effectiveness
Was our hypothesis valid?Was our hypothesis valid?Are we doing well?Are we doing well?
Inputs
Did we do the right thing in a worthwhile way?Did we do the right thing in a worthwhile way?
Three Major Components of Good DME
L
o
g
i
c
a
l
F
r
a
m
e
w
o
r
k
D
e
t
a
i
l
e
d
M
&
E
P
l
a
n
S
y
s
t
e
m
a
t
i
c
D
M
E
-
I
S
Levels of DME-IS (from recent DME-IS conference)
Level 1: paper-based, manual system
1
Information Requirements
L
e
v
e
l
o
f
s
o
p
h
i
s
t
i
c
a
t
i
o
n
23
44
Level 2: Common software programs, e.g. Excel, Word
Level 3: Database & analysis software
Level 4: Fully integratedsoftware package
We need to recognize that not all We need to recognize that not all projects are ready for level 4 projects are ready for level 4
DMEDME--IS. Even at level 1 IS. Even at level 1 (manual systems) there is room (manual systems) there is room
for improved systemization. for improved systemization.
In fact, systemize before you In fact, systemize before you automate needs to be the guiding automate needs to be the guiding
principle behind all DMEprinciple behind all DME--IS.IS.
The quality of DME in a project The quality of DME in a project can be greatly enhanced by using can be greatly enhanced by using
a fully integrated DMEa fully integrated DME--IS (like IS (like MER)MER), for it depends upon, , for it depends upon, complements, and makes complements, and makes
functional a logical logframe and functional a logical logframe and well detailed M&E plan. Not only well detailed M&E plan. Not only
for one project, but related for one project, but related projects within a larger program.projects within a larger program.
Over-all Principles of good DME
8Holistic diagnosis of needs and opportunities which includes community participation8Logical project designs8Systematized monitoring systems8Quality evaluations that measure impact, are credible and useful
8Decision makers think evaluatively
Whats meant by thinking evaluatively?4Reality checks: We all need to seek
objective feedback, gain perspectives on our work; learn lessons and apply them.4Rational decision-making: Before
making decisions be clear on what evidence we have, from whom it was obtained, and how reliable it is.
Whats meant by thinking evaluatively?4Challenge paradigms: -- Our own as
well as others. Be visionary. Think out of the box. Ask what other perspectives would be informative.
Whats meant by thinking evaluatively?4Be accountable: We owe it to our clients
(intended beneficiaries as well as donors) to document not only our use of inputs and production of outputs, but also what outcomes are achieved; i.e. what difference our projects have made in the lives of real people.
In these ways we can assure that In these ways we can assure that
all of us working togetherall of us working togetherwill help make the world a will help make the world a
better place for allbetter place for all !!!!
Nagarkot, Nepal; photo by Richard Caldwell
Introduction to the CARE InternationalDesign, Monitoring & Evaluation (DME) Standardsby Jim Rugh November, 2001Example for childhood malnutrition projectThe quality of each level is measured by the next higher level.ACCOUNTABILITYDIFFERENT PARADIGMSDIFFERENT PARADIGMSA little more on basic DME methodologies consistent with the DME StandardsAn Overview of the DME Cycle in CARESuggested criteria for determining appropriate design for a projects evaluation planSuggested criteria for determining appropriate design for a projects evaluation planOver-all Principles of good DMEWhats meant by thinking evaluatively?Whats meant by thinking evaluatively?Whats meant by thinking evaluatively?In these ways we can assure that
Top Related