VAMESP II Monitoring & Evaluation Elementary Opening and Introduction to the Course.
-
Upload
caren-gaines -
Category
Documents
-
view
213 -
download
0
Transcript of VAMESP II Monitoring & Evaluation Elementary Opening and Introduction to the Course.
VAMESP II
Monitoring & Evaluation
Elementary
Opening and Introduction to the Course
VAMESP II
Getting to know each other?
Who am I ? What is my background? What do I do now? What is my involvement
with Monitoring and Evaluation
What are my expectations from this workshop / training activity
Illustration by Petra RÖhr-Rowendaal
VAMESP II
Objective of these sessions Review principles of M&E and methodologies of
implementation – Building awareness that M&E is more than just data
collection and progress reporting M&E is an integral part of good practice in project
management. Setting M&E in the context of regulatory and donor
requirements. Collaborative working on practical M&E issues
(logframes, developing indicators, preparing/planning for monitoring) as they affect course participants.
Introduce the planning tools required for robust M&E.
VAMESP II
Content – what will we discuss?
What is monitoring and evaluation? What must we do for GoV and Donors? What information do managers need? Planning for M&E using Logframe
approach? Selecting indicators for M&E? Developing an M&E Plan? Communicating M&E information.
VAMESP II
What method will we use? Advantages of participatory
learning
Brief presentations Discussion and information sharing Practical exercises Using the M&E Manual as a resource
Demands a collaborative and participative approach
Illustrations by Petra RÖhr-Rowendaal
VAMESP II
Monitoring & Evaluation
Elementary – National M & E System
Section 2
Legislation, Principles and Elements
VAMESP II
Framework for M&E of ODA investments
National Strategic objectives for ODA are based on: 10 year social and economic strategy A 5 year development plan The ODA Master Plan Comprehensive Poverty Reduction and
Growth Strategy Focal point for M&E of ODA is FERD in
the Ministry of Planning and Investment
VAMESP II
M&E Responsibilities Legal basis in Chap VI, Decree
131/2006/ND-CP Various levels of responsibility:
of Project management units of Line Agencies of Ministry of Planning and Investment
Various types of reporting Monthly, quarterly, annual and terminal
reporting
VAMESP II
Good Practice in M&E
Good practice principles of M&E Reporing and monitoring systems country owned
and led Donors aim to rely on country owned reporting and
monitoring systems Reporting and monitoring systems support country’s
own information needs M&E systems should be simple and harmonised Information should be shared and transparent Evaluation missions should be planned and
implemented in partnership
VAMESP II
Principles of the Vietnam M&E system
Usefulness Impartiality and independence Credibility Participation Harmonisation Scheduled monitoring and evaluation program Professionally designed evaluations Cost effectiveness Report communicate & feedback results Using results to manage
VAMESP II
Elements of the Vietnam M&E system Glossary of M&E terms Purpose and design Indicators that are S M A R T Measures Tools and pro-formas Analysis and interpretation Reporting Sharing information
VAMESP II
Monitoring flow chart
Chủ DA
VAMESP II
National Monitoring System-Progress
Quaterly Progress AMT thought Project Owner
PMU PMU PMU PMU
Quaterly Progress AMT thought Project Owner
PMU PMU PMU PMU
Quaterly Portfolio
PMT
Line Agency
Line Agency
Donor Donor
Quaterly Portfolio
PMT
Feedback & Banchmarking
Feedback&Benchmarking
National Agencies
VAMESP II
PMU
Annual PlanDisbursement & Outputs
Monitoring data:Actual disbursement
Monitoring data:Actual outputs
Ø Monthly progress report to MPI, MoF & line agency for Gp A projectsØ Quarterly progress report to line agency, MPI and donorØ Quarterly technical report to related technical ministriesØ Bi-annual national outcome indicators to line agency, MPI and donor
Monitoring data:National outcome indicators
M&
E R
epor
t
ODA monitoring - PMU
VAMESP II
Line
age
ncy
(Min
istr
y or
Pro
vinc
e)
Aggregate annual PlanDisbursement & outputs
Aggregated data:Actual disbursement
Aggregated data:Actual outputs
Ø Quarterly summary progress reports to MPI, MoF and donorsØ Quarterly feedback report to PMUsØ Bi-annual provincial or sector national indicators to MPI and donors
Analysed data:National outcome indicators
M&
E R
epor
t
ODA monitoring – line agency
VAMESP II
ODA monitoring – MPIN
atio
nal O
DA
Man
agem
ent A
genc
ies
Ø Monthly summary exceptions report to PM for cabinet meetingØ Quarterly summary progress reports to PM and donorsØ Quarterly feedback report to line agencies, PMUs and donorsØ Quarterly lessons learned to national M&E websiteØ Bi-annual summary of national indicators to PM and CG meeting
Dat
a Ex
chan
ge &
OD
A W
ebsi
te
National ODA PlanDisbursement & outputs
Consolidated data:Actual disbursement
Consolidated data:Actual outputs
Consolidated data:National outcome indicators
VAMESP II
Monitoring & Evaluation
Elementary – Foundations
Section 3
Communication
VAMESP II
Communication process
ChannelMessage ReceiverSender
Context
The Communications Process
Feedback
VAMESP II
Communication Process To get your message across to others
Communication process involves Sender Message Channel Receiver Feedback Context
Message is successful when the sender and the receiver perceive it in same way
VAMESP II
Communication process – The Sender
To be an effective communicator Establish credibility Display knowledge of the subject Display knowledge of the audience Know the context of message delivery
Know your audience Messages will be mis-understood if you
do not understand who you are communicating to!!
VAMESP II
Communication process – The Message
Written, oral and non-verbal communications are affected by:
Sender’s tone Method of organisation Validity of the argument What is communicated and what is left out Your individual style
Messages have intellectual & emotional parts
Intellect gives us ability to reason Emotion allows us to present motivational appeals, which
change minds
VAMESP II
Communication process – The Channel
Messages are conveyed through channels: Verbal face-to-face meetings Telephone and video-conferencing Written – letters, emails, memos and
reports
VAMESP II
Communication process – The Receiver
Messages are delivered to an audience Message will prompt actions or reactions
Audience brings ideas and feelings to communication process These influence understanding of the
message A successful communicator considers
these before delivering the message
VAMESP II
Communication process – Feedback & Context
Audience – provides feedback Verbal and non-verbal reactions
Monitor feedback It measures if the audience is
understanding the message Context – The situation in which the
message is delivered Surrounding environment or broader
culture
VAMESP II
Removing barriers Communication barriers can emerge
at any stage of the communication process
Aim is to lessen frequency of these barriers at each stage
Communication should be: Clear Concise Accurate Well planned
VAMESP II
Removing barriers at all stages If lengthy, disorganised or contains
errors: messages will be misinterpreted
Poor verbal or body language: Confuses the message
Barriers in context – come from too much information too fast
Understand your audience culture – converse and deliver your message to the particular audience
VAMESP II
Communication Models
What is a communication model Drawings Charts Diagrams Pictograms Schematics Cartoons
VAMESP II
Shannon & Weaver communication model
VAMESP II
Monitoring & Evaluation
Elementary - Planning for M & E
Section 4
The Logframe, Monitoring Planning and Logistics
VAMESP II
Logframe and M&E
The Logical Framework (Logframe) is a project planning matrix and a tool of analysis used at all stages of the investment cycle.
VAMESP II
Logical Framework Logical Framework – Logframe
A basic tool for investment design, implementation and monitoring and evaluation
Analyses existing situation Creates a hierarchy of actions to achieve outcomes Identifies potential risks to achieve outcomes Establishes how outputs and outcomes will be
monitored and evaluated Monitors investments during implementation
A number of names: PDM, LFM, LFA, ZOPP (but basic elements are the
same)
VAMESP II
Some Logframe Terminology Investment description Narrative summary Goal Purpose Component objective Objective Expected outcomes Objective hierarchy Hierarchy of results Intervention logic
Results Outcomes Outputs Activities Inputs Assumptions Indicators Performance
indicators Means of verification Frequency
VAMESP II
Logframe Matrix
Description Verifiable Indicators Means of
Verification
Assumptions
Goal
Purpose
Outcomes(Results)
Outputs
Activities
Inputs
Logical Framework Matrix
VAMESP II
Logical Framework Matrix
Description Verifiable Indicators Means of
Verification
Assumptions
Goal
Purpose
Outcomes(Results)
Outputs
Activities
Inputs
Logical Framework Matrix
VAMESP II
Developing a Logframe Matrix Is an iterative “step –by-step” process
The description (narrative summary) The assumptions The indicators The means of verification
Look back and test the logic Modify and test again
VAMESP II
Matrix Logic
VAMESP II
Linking LFA and Evaluation Criteria
VAMESP II
Developing a Monitoring Framework
Basic pre-conditions
Guiding the investment strategy for results
Create a learning environment
Ensure effective operations
Develop and use an M&E system
VAMESP II
The key steps in monitoring planning
Five steps Establish the purpose and the scope Identify performance questions, information
needs and indicators Planning information gathering and organization Planning critical reflection processes and events Planning the necessary conditions and capacities
VAMESP II
Contents of a monitoring plan Purpose and scope Approach Revised Logical framework and indicators Management information system and
reporting Monitoring work plan and timing of activities Establishing conditions and capacities Monitoring budget Annexes
VAMESP II
Summary monitoring framework
Logframe Reference
Indicator
What We will measure
How We will measure
Who will measue
Frequency of measurement
How results will be reported
Goal
Purpose
Outcomes
Outputs
Activities
Inputs
VAMESP II
Maintaining monitoring quality
Update the system to meet changing needs
Usefulness – M&E System provides practical information
Feasibility – methods and procedures are realistic and cost effective
Propriety – M&E activities are conducted legally, ethically and regard for welfare of those affected by the results
Accuracy – M&E outputs provide accurate information to support effective and efficient management
VAMESP II
Monitoring & Evaluation
Elementary - Indicators
Section 5
Types of Indicators
SMART characteristics
VAMESP II
Selecting indicators for monitoring
Indicators are “signposts” of change Indicators establish the measures on which
the monitoring system may be built. To select indicators it is necessary to identify
what information is needed to answer the performance questions.
What are performance questions??
VAMESP II
Performance Questions A performance question helps focus
monitoring activities on what is necessary to know if the investment is performing as planned and why that is so. Ask this question
“What question would we need to answer to know the extent to which an investment is achieving the objective and to explain the success or failure”
VAMESP II
Performance questions are not complex!
Logical Framework
Basic Performance Questions
Inputs What resources has the ODA investment actually procured and used?
Activities
What has the ODA investment actually done?
Outputs What has the ODA investment delivered as a result of its activities?
Outcomes
What has been achieved as a result of the outputs?
Purpose What has been learned from implementation of the ODA investment than can contribute to improved implementation or to building knowledge.
VAMESP II
Criteria and Indicators
provide the objective and measurable basis on which the assessment of performance and achievement can be analysed and allow a rigorous assessment on which judgements can be based in drawing conclusions and recommendations from the monitoring exercise.
they should be developed in conjunction with the stakeholders involved (including participation from the community) and may incorporate both quantitative and qualitative assessments.
VAMESP II
Principles in defining and selecting indicators
Do not remain static
Should complement existing systems
Developed in partnership with stakeholders
Simple and Cost effective to measure
Linked across scale Practical and objectively verifiable
VAMESP II
Characteristics of indicatorsTo be fully effective and usable
indicators need to be SMART:
Simple Measurable Attributable Relevant Timely
VAMESP II
Characteristics of Indicators
To be strategically meaningful indicators need to be clearly specified in terms of:
Inputs
Outputs
Outcomes
Impact
VAMESP II
Types (Signposts) of Change
Type of Change Example of ChangeThe presence of something
Number of restored district health centres
The type of access to a new innovation or service
The level of use
The extent of an activity or coverage
The relevance of the new innovation
The quality of an innovation
The effort required to achieve change
VAMESP II
Key types of indicators Quantitative
Easily quantified – measured by defined numerical values
kilometers of road constructed Number of commune primary schools provided
Qualitative Not as easy to measure with numerical values.
Narrative descriptions may be necessary Degree of participation of women Minutes of farmers meetings Perceptions of well being
VAMESP II
Categories of indicators
Leading indicators Measuring outputs/outcomes and performance drivers
Process indicators Routine data and measures used for attributing results
Lagging indicators Measuring final outcomes / impacts
VAMESP II
How to define indicators Decide on what to measure and why ?
Specify for each activity, intended result, or goal
Decide what is to be measured The quantity……..how much? The quality ………what? The target group…….who? The time/period……..starting when and for how long? The place………where?
Check whether the indicator(s) describe the overall purpose, results and activities accurately, If not, other indicators will have to be added, new ones found, or proxy measures used.
VAMESP II
What, How, Who, When
Logframe Reference
Indicator
What We will measure
How We will measure
Who will measue
Frequency of measurement
How results will be reported
Goal
Purpose
Outcomes
Outputs
Activities
Inputs
VAMESP II
The essential questions?? What to monitor and why?
How to obtain relevant and worthwhile information?
Who may be expected to use it and for what purpose?
VAMESP II
Types of Change
Type of Change Example of indicatorThe presence of something Number of restored district health centres
The type of access to a new innovation or service
Number of primary-aged school children with access to commune primary schools
The level of use The frequency of traffic use of new commune road
The extent of an activity or coverage
The proportion of mothers in the poorest 20% of households with access to village health workers
The relevance of the new innovation
Do seed banks resolve a production constraint or not
The quality of an innovation The quality of health services from restored District health Centres
The effort required to achieve change
The labour required for a new soil management technique
VAMESP II
Monitoring & Evaluation
Elementary – Data for Monitoring
Section 6
Baseline Data and Collection Methods
VAMESP II
The Data Journey
Source: IFAD 2002
VAMESP II
Data and Information
Data is raw material and has no meaning on its own
Synthesis and analysis of data produces information
Information explains and provides lessons which emerge as knowledge
VAMESP II
Types of Data and analysis Quantitative data
Analysis involves calculations – totals, average numbers, percentages compared to plan or targets. Can be more complex frequency, regression and cost-benefit analysis.
Qualitative data Analysis is very different and more difficult
than quantitative data. Can involve opinions, photo-points, stakeholder perceptions and non-standard answers
VAMESP II
Methods of comparison
“Before” and “after” Data gathered at the beginning, mid-term and
at completion
“With” and “without” (control) Compare data from locations with investment
activities and from locations without investment activities
“Variance from plan” Compare the actual performance or results with
the planned performance or results
VAMESP II
Sample analytical tool to transform data
GoV Participants Meeting Competency Criteria
05
1015
2025
3035
4045
H1-03
H2-03
H1-04
H2-04
H1-05
H2-05
H1-06
H2-06
H1-07
H2-07
H1-08
H2-08
Program half year period
GoV
Sta
ff P
artic
ipan
ts
(No.
)
-10.0%
10.0%
30.0%
50.0%
70.0%
90.0%
110.0%
GoV
Sta
ff P
artic
ipan
ts
(%)
Total participating 1/
No. meetingcompetency criteria 2/
% competent
Target
VAMESP II
Example – Comparing Activities
Activities Conducted Jan 2004 - Dec 2006
0
10
20
30
40
50
60
H1-2004 H2-2004 H1-2005 H2-2005 H1-2006 H2-2006
VAMESP Half-Year Period
Num
ber o
f Act
ivitie
s
Formal Training
OTJ Training Events
PCC Meetings
PMC Meetings
S H Consultations
TWG Meetings
Workshops
Study tours
Field Visits
VAMESP II
Monitoring & Evaluation
Elementary - Reporting
Section 10
Audience, Communication, Presentation & Interpretation
VAMESP II
Communication of findings
Why communicate M&E results Many potential audiences:
Government Funding agencies Steering Committees Cooperating institutions Implementing partners Primary stakeholders
Funders and managers need information on impact Partners and stakeholders want to understand problems
in order to find solutions
VAMESP II
Sharing Results of M&E
Build value throughout the M&E process to arrive at an informative, useful ‘story’ that makes an effective contribution to development cooperation
VAMESP II
Sharing results of findings
Develop a strategy for sharing results that optimizes the benefits to be derived from the monitoring & evaluation
Meet the information needs of key audiences while respecting the organisation’s agenda and context for development
VAMESP II
Communication opportunities
Carefully consider all opportunities for communicating results and be creative in devising a strategy that enhances the potential for making a ‘real’ contribution both within the organisation and externally.
VAMESP II
Communication principles
Identify the audience (use an audience matrix)
Integrate communication into M&E plan
Consider communication an investment
Communication requires feedback
VAMESP II
Communication and M&E Planning
Source: IFAD 2002
VAMESP II
Audience Matrix
Types of Monitoring Information
Types of
Audiences
Progress
toward
goal
Achieve-
ments
Economic
Impacts
Follow-up
Decisions
Actions
Community Members H M M L M
Project Management Unit Staff
M H L H H
Provincial Agency M H L H M
Ministry H L H H M
Funding Agencies H M H H M
Other groups L M L L L
Priority Code H = High M = Medium L = Low
Derived from: IFAD (2002)
VAMESP II
Practical Presentation of Information
Clarity of message for specific finding Frequency of communication Timeliness of feedback Consider location of the audience Effective use of graphics for analysis Keep feedback sessions focused
VAMESP II
Communication Media
Written reporting Qtrly, 6 mthly, annual & mid term reports,
newsletters Oral reporting
Direct discussion, radio programs Visual display
Graphs, charts, maps, photographs, video, drama
Electronic communication Email, electronic newsletters, websites
VAMESP II
Example – Interpreting data – Baseline in Time Series
GoV Participants Meeting Competency Criteria
05
1015
2025
3035
4045
H1-03
H2-03
H1-04
H2-04
H1-05
H2-05
H1-06
H2-06
H1-07
H2-07
H1-08
H2-08
Program half year period
GoV
Sta
ff P
artic
ipan
ts
(No.
)
-10.0%
10.0%
30.0%
50.0%
70.0%
90.0%
110.0%
GoV
Sta
ff P
artic
ipan
ts
(%)
Total participating 1/
No. meetingcompetency criteria 2/
% competent
Target
VAMESP II
Example – Interpreting data – Variance from plan
Plan VS Actual Disbursement - VAMESP II (A$)
$0
$200,000
$400,000
$600,000
$800,000
$1,000,000
$1,200,000
$1,400,000
$1,600,000
$1,800,000
$2,000,000
J an Feb Mar Apr May J un J ul Aug Sep Oct Nov Dec J an Feb
Plan
Actual
VAMESP II
Example – Interpreting data – Comparing types of activities
Activities Conducted Jan 2004 - Dec 2006
0
10
20
30
40
50
60
H1-2004 H2-2004 H1-2005 H2-2005 H1-2006 H2-2006
VAMESP Half-Year Period
Num
ber o
f Act
ivitie
s
Formal Training
OTJ Training Events
PCC Meetings
PMC Meetings
S H Consultations
TWG Meetings
Workshops
Study tours
Field Visits
VAMESP II
Example – Interpreting data – Comparing progress against parameters
00.5
1
1.52
2.5
33.5
4Efficiency
Effectiveness
ImpactRelevance
Sustainability
VAMESP II
Monitoring & Evaluation
Elementary – Results Based Management
Section 12
Concepts and Definitions
VAMESP II
What is RBM
Results Based Management (RBM) is a management philosophy and approach that emphasizes development results in planning, implementation, learning and reporting.
VAMESP II
RBM needs participation Participation is an important ingredient
of RBM. Experience shows that participation improves the quality, effectiveness and sustainability of development actions.
RBM involves the participation of stakeholders in the: planning of projects & programmes; implementation of projects; monitoring and reporting of results.
VAMESP II
Results Based Management – Definition
Results Based Management is defined as: “A management strategyy focussing on
perfromance and achievement of outputs, outcomes and impacts “ (OECD-DAC 2002)
A “Result” is also defined by OECD-DAC as: “The output, outcome or impact [intended or
unintended, positive and/or negative] of a development intervention”
VAMESP II
Managing “Outputs” vs “Outcomes”
“Administrative” management Focuses on managing inputs,
activities and production of “outputs”
“Results based” management Focuses on management towards the
achievement of outcomes
VAMESP II
Comparing “administrative” and “results based” management
Administrative Results BasedDescription of situation before project start
Baseline data to define the situation before project start
Benchmarks for activities and outputs
Indicators prepared for outcomes
Collect data on imputs, activities and immediate outputs
Collect data on outputs – are they contributing to outcomes
Routine reporting on provision of inputs and outputs
More focus on perceptions of change among stakeholders
Directly linked to a discrete project
More qualitative & quantitiative analysis of progress to outcomes
Provides information on administrative, routine implementation and management issues
Project and strategic partners look at broader development effectiveness
VAMESP II
Good management Good management still requires:
The recording and monitoring of inputs and outputs; but
Recognises: That outcomes also must be monitored to
measure the success of a development activity. Outcome monitoring is a continual and systematic
process of collecting and analysing data to measure performance of ODA investments
VAMESP II
Key RBM Concepts RBM seeks to capture the process of change in the short,
medium and long-term. Development results are commonly linked together in a
results chain. Short-term results or outputs will contribute to medium-term results or outcomes which in turn will contribute to the long-term results or impact.
Completed activities are not short-term results or outputs. Outputs are the actual effects of completed activities.
RBM should identify the links between gender equality issues and the results the project seeks to achieve.
Beneficiary Reach refers to individual, groups or organizations affected by or benefiting from the results.
Results are influenced by the socio-economic context, gender analysis, the local capacity of stakeholders and organizations, the participation of stakeholders, available resources and the programme/project duration.