Título del Proyecto - clear-la.cide.edu 2_Workshop... · RE methods are generally adaptable to...
Transcript of Título del Proyecto - clear-la.cide.edu 2_Workshop... · RE methods are generally adaptable to...
Defining Traits Rapid Evaluations
Session 2
1
Session structure
Introduction – Rapid Evaluations…a new concept?
Objectives – Define REs – Specify particular traits of REs
Main topics – Definition – Key constitutive elements – Rationale, timing and users
Session Re cap
2
Previous session Re Cap (Evaluation, M&E and Rapid Evaluations for Government Programs)
The reasons for rapid evaluations
The concept of evaluation
Program Evaluation and Program Monitoring
M&E systems
Different Types of evaluations
Rapid Evaluations introduced
Rapid Evaluations and Information
3
Course Overview
1. Evaluation, M&E and Rapid Evaluations for Government Programs.
2. Defining Traits of Rapid Evaluations
3. Methods and techniques for Rapid Evaluation
4. Rapid Evaluations in practice I
5. Rapid Evaluations in practice II
6. Institutional context, application and use of Rapid Evaluations
4
Rapid Evaluations may be regarded as a subset of a larger family of evaluation methodologies, known in other fields as rapid appraisals, rapid assessments, desk reviews, etc.
The novelty of Rapid Evaluations is its systematic and more general use by government agencies in order to have better mechanisms for decision making.
Rapid Evaluations have benefited from the use of these methodologies in other fields or contexts.
Appraisals
The purpose is to develop a preliminary, qualitative understanding of a situation. Developed and applied mostly in anthropological studies.
Assessments
Research approaches that are intended to be efficient and cost effective; savings are found in reduced sampling and more efficient data analysis. Common in medical, environmental and biological sciences.
Desk Research (or Reviews)
Mostly used in the context of market research it refers to the use of secondary information or information readily available in order to conduct analyses of various topics.
Rapid… Rapid evaluations
5
Definining Rapid Evaluation
Starting point: working definition
We understand Rapid Evaluation (RE) as an evaluation approach that draws on different research methods and techniques and mostly on existing information to quickly, yet systematically, collect, analyze and synthesize evidence on key aspects related to public interventions, such as design, outputs, processes, outcomes and impacts.
Using RE is extremely useful in the face of important constraints of time and resources to provide valuable decision making inputs for budget, planning and to improve program performance.
6
Key constitutive elements of Rapid
Evaluations
Scope (content)
Information
Methods
Use
Budget & Time
7
Scope
A Rapid Evaluation:
Captures and assess the core traits of programs relevant for decision making.
Most commonly an RE:
– Examines the coherence of the program design
– Establishes connection between design and implementation
– Looks broadly at all the links of the results chain, from inputs to outputs to impacts.
– Reviews performance indicators
– Clarifies existing evidence on outcomes or impacts.
8
Scope
Always includes evaluation of program logic (design evaluation)
Plus some assessment of implementation (organization, process or management evaluation)
Plus a review of available performance or impact indicators (performance evaluation).
9
Information
Where does the information for RE come from?
A mix of data sources
– Documental information already available becomes a fundamental source.
• Documental information can come from a program's monitoring and information systems, previous studies, analysis or evaluations.
‒ Other types of information sources derived from:
• Observation
• Interviews
• Meetings
‒ Generally little or no additional systematic primary data collection
10
Methods
There is no unique strict method for REs
RE methods are generally adaptable to scope and context
– The scope of an RE depends on programs’ maturity, stage, or on specific information needs.
– Context refers to available information, development of M&E
systems and accountability needs.
11
Methods
But common to REs is the aim to systematically and analytically synthesize important programs´ information in order to be useful for accountability purposes and decision making.
The challenge is to know what information to consider, what aspect of a program to evaluate and how information gets analyzed and synthesis is developed.
12
Methods
REs use:
‒ Within a national, region or sector, a standardized methodology to analyze a set of programs.
‒ A homogeneous set of information to facilitate some (to the extent of possible) comparability across programs.
‒ Replicable instruments and processes.
REs aim is to
‒ Be applied to a significant number and a broad range of programs
‒ Be Implemented serially in the context of “systems of evaluation”
13
Methods
REs may apply ‒ Different types of methods, qualitative and quantitative, to analyze
a program ‒ Other methods to gather information, such as focus groups,
observation, interviews. ‒ A combination of analytic instruments, like the LogFrame matrix,
short questionnaires, etc.
REs do not
◦ Apply sophisticated quantitative methods on the basis of extensive primary data collection
◦ Draw conclusions on the impact and final outcomes on their own
14
Use
REs are intended as “decision making” tools
‒ Identify key messages and findings for decision making
‒ Contribute to recommendations formation
‒ Continuous program improvements
‒ Executive format and style of the results
15
Budget and Time
REs ‒ Are designed to be completed in a short period of time
Cost efficiency and time opportunity are a consequence of other constitutive dimensions ‒ Scope and methods
‒ Information used
‒ Coverage and scale
‒ Relevance for decision making
16
Key constitutive elements of Rapid Evaluations
• Decision makers at different levels
• Accountability
• Improvement of programs
• Budget decisions
• Mixed Data sources
• Documents/ Field
• Cost Effective
• Adaptable to context and scope
• Replicable
• Review
• Standardization
• Design/Logic framework
• Implementation (process, organization, planning)
• Results/outcomes Scope Method
Use Information
17
Rationale (for evaluation in general)
Two basic reasons for government:
“Internal”
– Need to enhance public sector performance
– Evidence based decision making
“External”
– Political accountability and transparency
18
Rapid Evaluations: Why?
Flexible and timely tool that allows to inform policy and budgeting decisions in the short run.
Generalising and transversal tool that permits to undertake a global comprehensive review of a program.
Standardized methodology that can be applied serially and allows obtain similar types or categories of performance information that facilitate the knowledge across programs
19
Rapid Evaluations: Why?
Manageable and adaptable tool that permits evaluating a number of programmes at the same time
Constructive methodology that permits to produce and consolidate information when lacking
…
20
Timing
Different aspects to consider about the timing of RE
– Country context and characteristics of M&E systems
– Policy/Program Cycle
– Program stage and maturity
21
Country context and characteristics of M&E systems
Aspects that are worth considering
– Degree of institutionalization of the government’s evaluation process
– Agencies and actors involved in the evaluation system
– M&E systems
– Evaluation capacity
22
Rapid Evaluations: When?
Projects/programs with….
– Upcoming key decision making points in the short run
– Mix of questions about design, process and performance (outputs, impacts)
– Generic lack of knowledge about performance
– Good quality secondary data
In preparation for a more focused operation or impact evaluation
….
23
Rapid Evaluations: When?
Timing of RE and the policy cycle
– When program has been
designed, or needs to be redesigned.
– When program is implemented
– After program implementation to assess results
– Periodical RE for continuous program improvement
• Rapid Evaluation
• Rapid Evaluation
• Rapid Evaluation
• Rapid Evaluation
(Re) Design Implement
ation
Results
Evaluation and
Budget Decisions
24
For whom?
Central Agencies
(Planners, budget
holders)
Legislators
Program Managers
(operators, implementators)
Civil Society Organizations
Citizens
25
For budget holders, central planning agencies
Pressure on agencies to perform (continuous improvement) – By reviewing their budget requests in the light of their past
performance (effectiveness and efficiency)
– By monitoring the implementation of recommendations and adjustment plans that have emerged from evaluations
Expenditure prioritization – Which programs which should be cut back, eliminated or expanded?
– Budget-linked performance targets
– Assess how far the agency is responsible for any failure to meet target
Accountability towards the general public
26
For program managers /implementers
Strategic directioning
– Improve program design and achieve higher outcome effectiveness
Internal management
– Improve programme implementation and achieve higher efficiency and cost effectiveness
Internal (within government) and external (towards citizens) accountability
27
REs for budget holders, central planning agencies
Evaluations timing must fit in with budget preparation
– Means evaluations usually need to be quick
– Results before final budget decisions taken
Similar types or categories of performance information allow a more comprehensible view across programs and facilitate some decision making
Uniform evaluation approach (metric)
28
RE for budget holders, central planning agencies
Evaluation reports designed to tell budget decision-makers what they need to know
– How effective the program is
– Whether it is inherently ineffective
– Or could be effective with better implementation
Focus on design evaluation permits questioning the reason of existence of programs
– Inform if the design is adecuate to resolve the problem
– Highlight issues of inter-institutional coordination
29
REs for program managers/ implementers
Integral assessment – Feedback that links design, implementation, outputs and
results
– Identify bottlenecks by looking at the “big picture” and highlighting “core” issues
Formative approach – Consolidates (non systematized) information in a unique
place
– Constructive approach in opposition to external “assessment” approach
30
Civil Society and Citizens
Increasing pressures for transparency and accountability
– There are increasing initiatives from Civil Society Organizations to analyze performance of governments
– Demands from citizens to know uses of public budget.
Importance of generating and facilitating access to quality information of government interventions
Transparency of evaluation process and evaluation by external agents give greater legitimacy to governments
31
Session ReCap
Definition of Rapid Evaluation
Key constitutive elements of Rapid Evaluations
– Scope
– Methods
– Information
– Use
– Timing
Rationale for RE
Users
32