Post on 20-May-2019
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
Monitoring and
Evaluation
Teen Club Community Partners Training Programme
Did you know …?Did you know …?
Managers commonly state that measuring Managers commonly state that measuring performance is too complex, time intensive, performance is too complex, time intensive,
and costly. and costly.
Many organizations consider M&E to be a Many organizations consider M&E to be a
requirement of the funding agencies that requirement of the funding agencies that support them and thus see M&E as an support them and thus see M&E as an external necessity. external necessity.
Few managers consider M&E to be the Few managers consider M&E to be the strategic system assessing organizational strategic system assessing organizational capacity, judging economic effectiveness or capacity, judging economic effectiveness or
predicting future programmatic sustainability. predicting future programmatic sustainability.
Principles and
purposes of monitoring, evaluation and
reporting
What Is Monitoring &
Evaluation?� Monitoring:
� Tracks information relevant to program planning, outputs, outcomes and impacts.
� Tracks costs & program functioning.
� Provides basis for program evaluation
� Evaluation:
� Is a rigorous, scientifically based collection of information about program activities, characteristics, and outcomes to determine the merit or worth of a specific program.
� Is used to improve programs and inform decisions about future resource allocations.
Purposes of Monitoring and
Evaluation
� Ensuring planned results are achieved
� Improving and supporting management
� Generating shared understanding
� Building the capacity of those involved
� Motivating stakeholders
� Ensuring accountability
� Fostering public and political support
Evaluative Thinking
Evaluation is a process and way of Evaluation is a process and way of thinkingthinking
Evaluation is not just about providing Evaluation is not just about providing information for someone else at the end of information for someone else at the end of
a projecta project
M & E should be an integral part of M & E should be an integral part of managementmanagement
M & E provide key information for M & E provide key information for managementmanagement
M & E can be creative, fun and rewardingM & E can be creative, fun and rewarding
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
Why M&E is Important
M&E provides managers with a M&E provides managers with a
tool to gain timely information on tool to gain timely information on the progress of project activities the progress of project activities
allowing them to compare what allowing them to compare what
was planned to what is actually was planned to what is actually occurring.occurring.
M&E promotes organizational M&E promotes organizational learning and encourages learning and encourages
adaptive management.adaptive management.
Why M&E is Important Cont.
M&E provides organizations with M&E provides organizations with
a tool to evaluate how a tool to evaluate how EFFECTIVELY they promote EFFECTIVELY they promote
change.change.
Essential Elements for a Successful Essential Elements for a Successful M & E Program M & E Program
Building an M&E system begins with Building an M&E system begins with
development of a Monitoring development of a Monitoring Evaluation and Reporting PlanEvaluation and Reporting Plan
An M&E plan documents data An M&E plan documents data
collection, evaluation and reporting collection, evaluation and reporting procedures to be undertaken by the procedures to be undertaken by the
organization. organization.
The M&E plan identifies:
� The mission and results sought
� Who needs information and data
from the organization, why, and
when.
� What data (management
information and indicators) will be collected by the organization.
� The source, method, frequency and schedule of data collection.
The M&E plan identifies (cont.):
Plans, schedules, and tools for evaluating Plans, schedules, and tools for evaluating information.information.
The team or individuals responsible for The team or individuals responsible for
data collection.data collection.
Plans, schedules, and tools for Plans, schedules, and tools for communicating information.communicating information.
How data will be analyzed, compared, How data will be analyzed, compared,
reviewed and presented, and the reviewed and presented, and the associated tools to be used associated tools to be used
Value of M&E for You Value of M&E for You
Facilitates project designFacilitates project design
Provides ongoing data and is thus a Provides ongoing data and is thus a
management toolmanagement tool
Provides evaluation data for reporting Provides evaluation data for reporting
purposespurposes
Can generate data for operational Can generate data for operational
researchresearch
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
The Project CycleThe Project CycleOrganisation
Mission
Financing and contracting
Final Evaluation
Gov/Donor Goals/Policies
Beneficiary Needs
Plan
Act
Monitor
Evaluate
Implementation
Monitoring and Evaluation
Scoping
Formulation / Design
M&E Strategy
Mobilisation and planning
M&E Plan
Results-Based M&E
The Development
Objective� Today organizations must present the
results of their programs as
contributing to a larger development objective.
� A development objective is the overall and long term effect of an intervention (it is your highest level of impact
anticipated). Reduction in incidence of HIV or improved food security are examples
Results Based Management (RBM)Results Based Management (RBM)An approach to project design and An approach to project design and management that emphasizes management that emphasizes
achieving results rather than simply achieving results rather than simply carrying out activitiescarrying out activities
Common sense but many projects Common sense but many projects having fallen into the trap of losing having fallen into the trap of losing sight of what they are really trying to sight of what they are really trying to
achieveachieve
RBM implies an effective project RBM implies an effective project monitoring and evaluation systemmonitoring and evaluation system
Setting Goals and
Objectives
DefinitionsDefinitions
Goal: a general statement that Goal: a general statement that
describes the hopeddescribes the hoped--for result of a for result of a programmeprogramme. Goals are achieved over . Goals are achieved over
the long term (5the long term (5--10 yeas) and often 10 yeas) and often
through the combined efforts of through the combined efforts of multiple multiple programmesprogrammes (e.g. reduction (e.g. reduction
of HIV/AIDS incidence)of HIV/AIDS incidence)
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
DefinitionsDefinitions
Objective: the specific, Objective: the specific, operationalizedoperationalized
statement detailing the desired statement detailing the desired accomplishment of the accomplishment of the programmeprogramme. .
Objectives address questions of what Objectives address questions of what and when but not why. and when but not why.
They should be stated in terms of They should be stated in terms of results to be achieved, not processes results to be achieved, not processes
or activities to be performed. or activities to be performed.
SMART MethodSMART MethodSpecific: Specific: Identifies concrete events or actionsIdentifies concrete events or actions
that will take place.that will take place.
Measurable: Quantifies the amount of resources,Measurable: Quantifies the amount of resources,
activity or change.activity or change.
Appropriate: Logically relates to the overallAppropriate: Logically relates to the overall
problem statement and desiredproblem statement and desired
effects of the program.effects of the program.
Realistic: Provides realistic dimension that Realistic: Provides realistic dimension that
can be achieved with availablecan be achieved with available
resources resources
TimeTime--based: Specifies a time within which the based: Specifies a time within which the
objective will be achieved.objective will be achieved.
Sample ObjectivesSample Objectives
Process Objective: Process Objective:
Provide VCT clients with HIV test Provide VCT clients with HIV test
results.results.
Outcome Objective: Outcome Objective:
Assist VCT clients with Assist VCT clients with
developing personalized riskdeveloping personalized risk--
reduction and treatment reduction and treatment strategies.strategies.
Examples of SMART ObjectivesExamples of SMART Objectives
Process Objective: Process Objective:
By the end of the first program year, By the end of the first program year, 98% of clients tested for HIV will 98% of clients tested for HIV will receive test results. receive test results.
Outcome Objective:Outcome Objective:
By the beginning of the second By the beginning of the second
program year, 65% of clients receiving program year, 65% of clients receiving HIV test results will have developed HIV test results will have developed and adhered to personalized riskand adhered to personalized risk--
reduction/treatment strategies.reduction/treatment strategies.
Examples of Objectives Improve use of TB register data (as measured by decrease Improve use of TB register data (as measured by decrease
in the transferin the transfer--out rate, interruption rate & missing data, as out rate, interruption rate & missing data, as well as improved validity of data) in 8 of 9 provinces by well as improved validity of data) in 8 of 9 provinces by
March 31, 2004.March 31, 2004.________________________________________________________________________________________________________
Pilot and evaluate a WindowPilot and evaluate a Window--based Electronic TB Register (ETR) in based Electronic TB Register (ETR) in 1 province by 3/31/03.1 province by 3/31/03.
Decrease the transferDecrease the transfer--out rate to < 7% in 8 provinces within 12 out rate to < 7% in 8 provinces within 12
months of implementation of ETR (baseline 2000, 11.1%).months of implementation of ETR (baseline 2000, 11.1%).
Decrease the interruption rate to <10% in districts within 12 months Decrease the interruption rate to <10% in districts within 12 months of implementation of ETR (baseline 2000, 15.1%).of implementation of ETR (baseline 2000, 15.1%).
Decrease missing data to less than 10% within 12 months.Decrease missing data to less than 10% within 12 months.
Conduct TA visits in 8/9 provinces and support implementation of Conduct TA visits in 8/9 provinces and support implementation of
the ETR particularly regarding validation and use of data.the ETR particularly regarding validation and use of data.
ExerciseExercise
Mafikeng OVC Trust is developing a Mafikeng OVC Trust is developing a
satellite Teen Club satellite Teen Club programmeprogramme in in order to increase access to care and order to increase access to care and
support for HIVsupport for HIV--positive adolescents positive adolescents
in Northwest Province.in Northwest Province.
Write specific SMART objectives for Write specific SMART objectives for
the the programmeprogramme
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
Logic Model
Logic Model Definition
A logic model describes the main elements of a A logic model describes the main elements of a
program and how they work together to prevent program and how they work together to prevent
HIV in a specific population.HIV in a specific population.
This model is often displayed in a flow chart, map This model is often displayed in a flow chart, map
or table to portray the sequence of steps leading or table to portray the sequence of steps leading
to program outcomes. to program outcomes.
Levels of Results, Inputs Levels of Results, Inputs
& Processes& Processes
Inputs & ProcessesInputs & Processes
OutputsOutputs
OutcomesOutcomes
Impact Impact
Levels of Results - Outputs
� Outputs are information, products, or results produced by undertaking activities
� Outputs relate to completion of activities.
� Outputs reflect what you hoped to produce
from a particular input. For example: You decide the process you want to use is to train people, thus people trained is the result at
the input/process level
� Outputs usually reflect a result achieved in a
relatively short-time period (0-2 years).
Levels of Results - Outcomes
Outcomes are broad changes in Outcomes are broad changes in development conditions, answering the “so development conditions, answering the “so
what” question (So… we trained 100 people what” question (So… we trained 100 people and increased their knowledge but did or did and increased their knowledge but did or did they not change their behavior?). they not change their behavior?).
Outcomes often reflect behavior or economic Outcomes often reflect behavior or economic change and show how our activities change and show how our activities
contribute toward development objectives. contribute toward development objectives.
Outcomes usually reflect a result achieved Outcomes usually reflect a result achieved an intermediate time period (2an intermediate time period (2--5 years) 5 years)
Levels of Results - Impacts
Impacts are the overall and long term Impacts are the overall and long term effects of an intervention. Impacts are effects of an intervention. Impacts are
the ultimate result attributable to a the ultimate result attributable to a development intervention over an development intervention over an extended period. Improved food extended period. Improved food
security, increased standards of living security, increased standards of living etcetc
Impacts usually reflect a result Impacts usually reflect a result achieved over a longer time period (5achieved over a longer time period (5--10+ years)10+ years)
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
A Good Logic Model
� Includes a problem statement, inputs, activities, outputs, outcomes, and impacts
� Reflects agreement among major stakeholders about intended implementation
and outcomes (planned logic model)
� Illustrates clear, sequential, and logical linkages between each part of the logic
model
� Contains a problem statement that identifies
underlying causes
VCT Program Logic Model
Counseling and Testing
Personnel*
Funding from gov’t, GAP, & other donors
VCT MIS**
HIV test kits
Referral system for
prevention & Tx services**
ACTIVITIES OUTPUTS OUTCOMES IMPACTSINPUTS
VCT protocols,guidelines, and
training documents**
Problem Statement: HIV infection rates continue to rise, underscoring the importance for people to know their serostatus:
develop personalized risk-reduction strategies and access care and treatment services.
ExerciseExercise
Write down the inputs available Write down the inputs available
for implementation of the satellite for implementation of the satellite
Teen Club Teen Club programmeprogramme for for MafikengMafikeng
VCT Program Logic Model
Train counseling & testing personnel and site managers
Counseling and Testing
Personnel*
Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested
Provide pregnant HIV+ women with PMTCT services
Funding from gov’t, GAP, & other donors
VCT MIS**
HIV test kits
Referral system for
prevention & Tx services**
Screen HIV+ clients for (or refer to
screening for) OIs and TB specifically
ACTIVITIES OUTPUTS OUTCOMES IMPACTSINPUTS
Refer HIV+ clients to Tx services
VCT protocols,guidelines, and
training documents**
Exercise Exercise
Define the activities the Mafikeng Define the activities the Mafikeng
OVC Trust should undertake to OVC Trust should undertake to
implement the Mafikeng satellite implement the Mafikeng satellite Teen Club Teen Club programmeprogramme..
VCT Program Logic Model
Train counseling & testing personnel and site managers
Counseling and Testing
Personnel*
Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested
Provide pregnant HIV+ women with PMTCT services
Funding from gov’t, GAP, & other donors
VCT MIS**
HIV test kits
Referral system for
prevention & Tx services**
Screen HIV+ clients for (or refer to
screening for) OIs and TB specifically
# clients received pre-test counseling,# clients received HIV tests,# clients received results & post-test counseling***
# pregnant HIV+ women receive PMTCT services
# HIV+ clients referred to or
provided OI/TB screening
# personnel certified in VCT
ACTIVITIES OUTPUTS OUTCOMES IMPACTSINPUTS
Refer HIV+ clients to Tx services
# HIV+ clients referred to Tx, care & support including
ARVs
VCT protocols,guidelines, and
training documents**
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
ExerciseExercise
Determine the outputs (i.e. shortDetermine the outputs (i.e. short--
term results) you expect from the term results) you expect from the
new Mafikeng satellite Teen Club new Mafikeng satellite Teen Club programmeprogramme
VCT Program Logic Model
Train counseling & testing personnel and site managers
Risk behaviors decreased
Counseling and Testing
Personnel*
Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested
Provide pregnant HIV+ women with PMTCT services
Funding from gov’t, GAP, & other donors
VCT MIS**
HIV test kits
Referral system for
prevention & Tx services**
Screen HIV+ clients for (or refer to
screening for) OIs and TB specifically
# clients received pre-test counseling,# clients received HIV tests,# clients received results & post-test counseling***
# pregnant HIV+ women receive PMTCT services
# HIV+ clients referred to or
provided OI/TB screening
Quality of VCT increased
Access to VCT increased
Clients (HIV+ and -) develop & adhere to
personalized HIV risk-reduction &
treatment strategy
Increase in care, prevention, and
treatment services for HIV+, HIV-, and
discordant couples
# personnel certified in VCT
ACTIVITIES OUTPUTS OUTCOMES IMPACTSINPUTS
Refer HIV+ clients to Tx services
# HIV+ clients referred to Tx, care & support including
ARVs
Health outcomes of HIV + improve
VCT protocols,guidelines, and
training documents**
ExerciseExercise
Determine the outcomes (i.e. Determine the outcomes (i.e.
midmid--term results) you expect from term results) you expect from
the new Mafikeng satellite Teen the new Mafikeng satellite Teen Club Club programmeprogramme..
VCT Program Logic Model
Train counseling & testing personnel and site managers
Risk behaviors decreased
Counseling and Testing
Personnel*
Provide pre-test counseling, HIV testing, and post-test counseling to all clients tested
Provide pregnant HIV+ women with PMTCT services
Funding from gov’t, GAP, & other donors
VCT MIS**
HIV test kits
Referral system for
prevention & Tx services**
Screen HIV+ clients for (or refer to
screening for) OIs and TB specifically
# clients received pre-test counseling,# clients received HIV tests,# clients received results & post-test counseling***
# pregnant HIV+ women receive PMTCT services
# HIV+ clients referred to or
provided OI/TB screening
Quality of VCT increased
Access to VCT increased
Clients (HIV+ and -) develop & adhere to
personalized HIV risk-reduction &
treatment strategy
Increase in care, prevention, and
treatment services for HIV+, HIV-, and
discordant couples
HIV transmission rates decreased
HIV incidence decreased
HIV morbidity & mortality decreased
# personnel certified in VCT
ACTIVITIES OUTPUTS OUTCOMES IMPACTSINPUTS
Refer HIV+ clients to Tx services
# HIV+ clients referred to Tx, care & support including
ARVs
Health outcomes of HIV + improve
VCT protocols,guidelines, and
training documents**
ExerciseExercise
Determine the impact (i.e. longDetermine the impact (i.e. long--
term results) you expect from the term results) you expect from the
new Mafikeng satellite Teen Club new Mafikeng satellite Teen Club programmeprogramme.. Indicators
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
What is an indicator?
� Indicators identify what you will measure to know if conditions have or have not
changed.
� An indicator is specific information that
provides evidence as to the achievement of results and activities
� Indicators can be quantitative measures or
qualitative (narrative related) observations
� Indicators are indicative – it enables you to
reduce a large amount of data down to its simplest form
Why are Indicators Important?
� Indicators serve as tools to examine trends and highlight problems.
� Indicators provide an early warning signal to predict future conditions.
� When compared with targets, indicators can signal the need for corrective management action, can evaluate the
effectiveness of various management actions, and provide evidence as to whether objectives are being achieved.
Characteristics of Good Indicators
Measurable
Practical (data can be collected on a timely basis and at reasonable cost).
Reliable (can be measured repeatedly
with precision by different people).
Relevant - Attributable to YOUR
ORGANZATION (the extent to which a result is caused by YOUR activities).
Management Useful (the information provided by the measure is critical to decision-making).
Characteristics of Good Indicators
Continued
Direct (the indicator closely tracks the result it is intended to measure).
Sensitive (serves as an early warning of changing conditions).
Responsive (can be changed by management control).
Objective (the measure is operationally precise).
Exercise Exercise
Make a list of possible indicators Make a list of possible indicators
associated with the outcomes you associated with the outcomes you
have determined for the new have determined for the new Mafikeng satellite Teen Club Mafikeng satellite Teen Club
programmeprogramme
Selecting the “Best” IndicatorsSelecting the “Best” Indicators
� Based on your analysis, narrow the list to the final indicators that will be used
in the monitoring system.
� They should be the optimal set that
meets management needs at a reasonable cost.
� Limit the number of indicators used to
track each objective or result to a few (two or three).
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
Data Collection,
Analysis and Dissemination
Data Collection and StorageData Collection and Storage
The data collection methods The data collection methods
including standardized data including standardized data collection tools, explanatory notes collection tools, explanatory notes
and the means for data and the means for data
aggregation (e.g. cumulative, aggregation (e.g. cumulative, monthly report forms) should be monthly report forms) should be
detailed in the M&E plan. detailed in the M&E plan.
Data Analysis and ReportingData Analysis and Reporting
Information provided by staff at all Information provided by staff at all
levels should be input in levels should be input in appropriate databases or entered appropriate databases or entered
in a filing system in a filing system
Data disseminationData dissemination
Data collected through M&E activities will Data collected through M&E activities will be used to highlight successful strategies, be used to highlight successful strategies,
model practices that can be replicated, model practices that can be replicated, and inform others about lessons learnt. and inform others about lessons learnt.
Information is also used to determine if Information is also used to determine if planned planned programmeprogramme objectives are objectives are achieved or not, if target populations have achieved or not, if target populations have
been reached and the desired quality of been reached and the desired quality of services provided has been reached.services provided has been reached.
Quality feedback loopQuality feedback loop
�� Failure to reach preset goals Failure to reach preset goals
prompts investigation of possible prompts investigation of possible causes.causes.
Quality element of BIPAI M&EQuality element of BIPAI M&E
Feedback on aggregated dataFeedback on aggregated data
according to defined schedule andaccording to defined schedule and
preset criteria: preset criteria:
e.g. 3 month adherence data on alle.g. 3 month adherence data on all
patients recently initiated on AVRs:patients recently initiated on AVRs:
failure to attain a level of 80% offailure to attain a level of 80% of
patients at or above 95%patients at or above 95%
adherence is flaggedadherence is flagged
Monitoring and Evaluation Capacity Building for Program Improvement
October 28-31, 2003
Quality element of Quality element of
BIPAI M&EBIPAI M&E
Immediate feedback on a per Immediate feedback on a per
patient basis according to preset patient basis according to preset
criteria: criteria:
e.g. patient is flagged if not e.g. patient is flagged if not
reaching desired efficacy on reaching desired efficacy on ARVs assessed as assessed by ARVs assessed as assessed by
viral loadviral load
Evaluation
Key Evaluation QuestionsKey Evaluation Questions
� Relevance - Was/is the project a good idea given the situation to improve? Was
the logic of the project correct? Why or Why Not?
� Effectiveness - Have the planned results been achieved? Why or Why Not
� Efficiency - Have resources been used in
the best possible way? Why or Why Not?
Key evaluation questions (cont.)Key evaluation questions (cont.)
Sustainability Sustainability -- Will there be continued Will there be continued positive impacts as a result of the project positive impacts as a result of the project
once it has finished? Why or Why Not?once it has finished? Why or Why Not?
Impact Impact -- To what extent has the project To what extent has the project contributed towards its longer term goals? contributed towards its longer term goals?
Why or Why Not? Have there been any Why or Why Not? Have there been any unanticipated positive or negative unanticipated positive or negative consequences of the project? Why did consequences of the project? Why did
they arise?they arise?
ExerciseExercise
Formulate the key evaluation Formulate the key evaluation
questions you would like to be questions you would like to be
answered at the midanswered at the mid--point or point or completion of the new Mafikeng completion of the new Mafikeng
satellite Teen Club satellite Teen Club programmeprogramme..
M&E Data Use
ProgramImprovement
ProgramImprovement
Reporting/Accountability
Reporting/Accountability
Share
Data withPartners
Share
Data withPartners