Build Your NGO: Monitoring & Evaluation
-
Upload
allie-hoffman -
Category
Technology
-
view
8.673 -
download
0
description
Transcript of Build Your NGO: Monitoring & Evaluation
Monitoring & Evaluation
Agenda
• Introduction• Basic concepts
Definitions of M&EWhy do M&E?
• Project PlanningTheory of changeResult chainAssumptions and risks
• M&E– Introduction to M&E
systems– How to measure
• What indicators to use
• How to collect data• What targets to set
– Putting it all together
Definitions
• MonitoringRoutine, systematic collection of data relating to project
performanceProvides regular feedback and early indications of
progress (or lack of progress)
• EvaluationEpisodic and objective assessment of projectOften done at mid-point and at end of projectCan determine whether objectives were achieved, how
efficient the project was, and what the project’s impact was
Why do M&E?
• Knowledge of impactM&E tells you whether your projects are meeting
objectives and having results• Improved decision-making
Information generated through M&E helps you make better decisions about projects
• Greater accountability and transparencyM&E data allows donors and others to assess how well and
efficiently you are using resources
Theory of Change
• What is the theory of change for your project?• A chain of so thats that lead to the change you hope
for• Example: we teach children how to wash their hands
So that they will understand the importance of washing their hands AND will have the skills to wash their hands properly
So that they will wash their hands regularly So that they will be less susceptible to certain illnessesSo that the incidence of certain illnesses will decrease So that child mortality will decrease
Result Chain
Def: A picture of the chain of “so that” relationships that lead to the desired
outcomes
Source: University of Wisconsin Extension, Developing a Logic Model
Result Chain – Simplest Form
Inputs Activities Outputs
Purpose/ Short-term
objective
Goal/ Long-term
objective
Result Chain – More Detailed
Inputs
•What you invest
•Staff•Volunteers•Time•Money•Research base
•Materials•Equipment•Technology•Partners
Activities
•What you do•Specific actions you undertake to achieve particular outputs
Outputs
•What you deliver•Products and/or services you deliver (~1 yr)•You should be able to guarantee these•Examples:•Conduct workshops•Deliver services•Develop products/curriculum/resources•Train•Provide counseling•Assess•Facilitate•Partner
Purpose/ Objectives
•What the short- and medium-term results are
•Situations, conditions, behaviour changed as a consequence of the project (~3-5 yrs)
•Outside your control
•Learning:•Changes in awareness, knowledge, attitudes, skills, opinions, aspirations, motivations
•Action:•Changes in behaviour, practice, policies, decision-making,
Goal / Long-Term Objective
•What the ultimate impact is
•Long-term impact of the project (+ 5 years)
•Project contributes to goal, but likely doesn’t achieve it
•Conditions:•Changes in social, economic, civic, environmental conditions
Result Chain Example: HIV Prevention for Sex Workers
Inputs
•Staff - 3 trainers
•Volunteers – national network of 8 volunteers
•Time – 20 hours/month
•Money – space rental, snacks purchase, transportation reimbursement
•Research base – knowledge of STDs, knowledge of training techniques
•Materials – workshop materials
Activities
•Make schedule of workshops
•Coordinate with local volunteers to recruit workshop participants
•Plan workshop curriculum
•Assemble workshop materials
•Book workshop facilities
Outputs
•Conduct 1 workshop every 2 months
•Target Cambodian sex workers
Purpose/Objectives
•Learning:•Participants are aware of the importance of HIV prevention
•Participants have knowledge of means of transmission, methods of prevention
•Participants have negotiation skills
•Action:•Participants practice safer sex
Goal/Long-Term Objective
•HIV prevalence among Cambodian sex workers declines
Assumptions and Risks
• How do you plan for events, conditions, and decisions beyond your direct control?
• Assumptions positive statements about what will go right
• Risksnegative statements about what might go wrong
Source: AusAid, AusGuideline 3.3 The Logical Framework Approach, 2005; NORAD, The Logical Framework Approach (LFA): Handbook for objectives oriented planning (4th ed.), 1999.
Assumptions and Risks
Inputs Activities Outputs
Purpose/Short-term
objective
Goal/Long-term
objective
Decreasing controlIncreasing significance of risks
Assumptions Assumptions Assumptions Assumptions
Decreasing controlIncreasing significance of risks
Models and M&E
• A result chain says what results you expect to achieve• M&E tells you if you are achieving the results you expected
to achieve
Good M&E Systems
• Dynamicencourage `learning by doing’
• Participative and gender sensitiveseek to overcome barriers of gender, age, power, and
culture• Reflective
encourage creating regular space and time for analysing information and reflecting on the theories of change
• Evolvingadapting and changing in order to keep them as light and
simple as possible while providing `real time’ information
Source: AusAid, Guidance on M&E for Civil Society Programs, December 2008
How to Measure: Indicators
• Indicators tell you how you will recognize success• Indicators are a unit of measurement
Quantitative Indicators
• Need a balance between quantitative and qualitative indicators
• Quantitative: Can be measured in numbers
number of water pumps installedamount of rice harvestedpercentage of children immunizedHIV prevalence rate student-teacher ratiofrequency of attending class
Qualitative Indicators
• Qualitative: reflect people’s judgments, opinions, perceptions, and attitudes
perception of well-beingappropriateness of interventionquality of engagementlevel of commitmentsense of empowerment
Indicators to Use
1) What questions do you want to answer? (i.e. your Key Performance Questions)?2) What data can provide the evidence you need to answer your KPQs? Is the data high-quality? Is it participative?3) What data can you collect? Do the benefits justify the costs?
Indicators to Use
1) What questions do you want to answer? (i.e. your Key Performance Questions)?• This depends on
Your audience– Management? Donors? Beneficiaries?
The purpose of measurement– To assess impact? Efficiency? Equity?
Conflicts can arise between M&E for accountability and M&E for learning
Indicators to Use
Level in result chain
Key Performance Questions - examples
Inputs • What interventions and resources are needed?• Are resources being used efficiently?
Activities • What are we doing? Are we doing it right?• Have planned activities been completed on time and within the budget?
Outputs • Are we implementing the project as planned?• What direct tangible products or services has the project/programme delivered?
Purpose/ objectives
• Are interventions working or making a difference?• What changes have occurred as a result of the outputs? Are these changes contributing towards the
project purpose and desired impact? • Has the project achieved the changes for which it can realistically be held accountable?
Goal/ long-term objective
• Are we intervening on a large enough scale?• To what extent has the project contributed towards its longer term goals? Why or why not? • What unanticipated positive or negative consequences did the project have? Why did they arise?
Sources: International Federation of Red Cross and Red Crescent Societies, Monitoring and Evaluation in a Nutshell, 2007; Global Fund, Monitoring and Evaluation Toolkit, 2009
Indicators to Use
2) What data can provide the evidence you need to answer your KPQs? Is the data high-quality? Is it participative?
Indicators to Use
Level in result chain
What to monitor & evaluate/ Key Performance Questions
Key Performance Indicators - examples
Inputs • Are resources being used efficiently? • Number of staff hours spent• Money spent
Activities • Have planned activities been completed on time and within the budget?
• What unplanned activities have been completed?
• Percent of milestones achieved on time
• Variance between budget and actual
Outputs • What direct tangible products or services has the project delivered?
• Number of workshops held• Number of sex workers trained
Purpose/ objectives
• What changes have occurred as a result of the outputs? To what extent are these likely to contribute towards the project purpose and desired impact?
• Has the project achieved the changes for which it can realistically be held accountable?
• Percent of workshop participants demonstrating a high level of knowledge of HIV transmission and prevention
• Percent of workshop participants reporting using condoms “frequently” or “always”
Goal/ long-term objective
• To what extent has the project contributed towards its longer term goals? Why or why not?
• What unanticiplated positive or negative consequences did the project have? Why did they arise?
• HIV prevalence rate among Cambodian sex workers
Indicators to Use
Ensure that high-quality data will be collected. It must be• Accurate• Reliable• Precise• Complete• Timely• With Integrity• Respectful of confidentiality
Source: Global Fund, Monitoring and Evaluation Toolkit, 2009
Indicators to Use
3) What data can you reasonably collect?
Indicators to Use
Inputs Activities Outputs
Purpose/Short-term
objective
Goal/Long-term
objective
Increasing difficulty of measurementDecreasing frequency of measurement
Result Chain
Indicators to Use
Increasing difficulty of measurement
Hierarchy of Effects
Participation
•Number and frequency of people reached
•Intensity of contact
Reactions
•Degree of satisfaction with programme
•Level of interest
•Feelings toward activities, educational methods
Learning
•Changes in knowledge, attitudes, skills, aspirations
Actions
•Changes in behaviours and practices
Social-economic-environmental Improvements
•Changes in society
Source: Bennett and Rockwell, 1995, Targeting Outcomes of Programs
Indicators to Use
• Do the benefits justify the costs?• Look for a balance between rigour and realism
Source: Results-Based Management Tools at CIDA: A How-to Guide, http://www.acdi-cida.gc.ca
How to Collect Data
Identify for each Performance Indicator:
Definition Examples
Data source(s) People, organisations, documents providing data about the indicator
Beneficiaries, partners, government documents
Data collection methods/tools
How data will be collected Analysis of documents, surveys, interviews, focus groups, observation, case study, tests, testimonials, expert panel
Formula/ scale / assessment method
How the data will be captured
Different scales
Frequency and duration
How often data will be collected and for how long
Single time, continuously, weekly/monthly/annually, baseline/mid-term/final
Responsible person(s)
Who will collect the data Staff, outside evaluators
Indicators to Use
• Indicators ≠ Targets• Indicators are units of measurement
E.g. HIV prevalence rate among Cambodian sex workers• Targets specify a particular value for an indicator to
be accomplished by a specific dateE.g. HIV prevalence rate among Cambodian sex workers is
reduced by 50% by 2020• Targets are essential: they are a concrete expression
of your goals and objectives
Targets to Set
• You can only set targets once you have baseline data• Baseline data show the situation before you start the
project
Targets to Set
• Targets can be set as Absolute targets (e.g. increase by 5)Proportional or percentage targets (e.g. increase by 5%)Relative to benchmarks (e.g. be within the top three
schools in our area)Relative to costs or budgets (e.g. increase or reduce by 5%
same level of budget).
Sources: Advanced Performance Institute, How to Design Key Performance Indicators, 2010; Results-Based Management Tools at CIDA: A How-to Guide, http://www.acdi-cida.gc.ca
Targets to Set
• Targets should be SMARTSpecific, Measurable, Attainable, Relevant, Time boundAmbitious but realistic
Targets to Set
• Targets should account forTrends and past performancePredictable variation in performance (e.g. seasonal cycles)National targets, best practice benchmarks, etcResult chain (i.e. do not set outcome targets before you
have set input targets)Time lag between intervention and effectDependence on others (e.g. partners, government actors)
Targets to Set
• Indicator - # of workshops held• Add target group (for whom?) - # of workshops for sex
workers• Add quality (what? how well?) - # of workshops for sex
workers addressing HIV prevention• Add quantity (how much?) - # of workshops for sex workers
addressing HIV prevention increased from 10 to 12• Add time (by when?) - # of workshops for sex workers
addressing HIV prevention increased from 10 to 12 in 2012• Add place (where?) -# of workshops for sex workers
addressing HIV prevention increased to 6 in Phnom Penh and 6 in the provinces in 2012
Source: NORAD, The Logical Framework Approach (LFA): Handbook for objectives oriented planning (4th ed.), 1999.
Putting it All Together…
Key Performance Questions
Key Performance Indicators
Baseline Target Means of Verification
Input Are resources being used efficiently?
# staff hours/workshop 90 in 2011 50 in 2012
Staff time sheets
Activities Have planned activities been completed on time and within the budget?
% variance between budgeted and actual spending
2012 budget
2012 budget
Records kept by accountant
Outputs What direct tangible products or services has the project/programme delivered?
# workshops held
# sex workers reached
10/year in 2011
300/year in 2011
12/year in 2012
480/year in 2012
Participant lists
Purpose/Objectives
What changes have occurred as a result of the outputs and to what extent are these likely to contribute towards the project/programme purpose and desired impact?
Has the project/programme achieved the changes for which it can realistically be held accountable?
% of workshop participants demonstrating a high level of knowledge of HIV transmission and prevention
% of workshop participants reporting using condoms “frequently” or “always”
50% before the workshop
5% before the workshop
90% after the workshop
20% after 1 month10% after 6 months
Questionnaires before and after workshop
Surveys before workshop, after 1 month, and after 6 months
Goal/Long-term objective
To what extent has the project/programme contributed towards its longer term goals?
HIV prevalence rate among Cambodian sex workers
X% n 2011 Y% 2020 UNAIDS statistics
Want to Keep Going?
Put Pari’s M&E Services to work for your organization!
• We can help you set up an M&E system from scratch or assess your current system and help you improve it
• We’ll asses your team’s capacity and make an inventory of current monitoring, evaluation and reporting activities
• We’ll work with your team to map out your programmes/projects, select indicators, and create tools and schedules for tracking data and engaging in evaluation
• We’ll help you pilot the new system, gather baseline data and set targets
• We’ll work alongside your team to maintain the system
Email us at [email protected] to learn more