Evaluation in Latin America. Evaluation for Transparency ... · evaluations, capacity building...
Transcript of Evaluation in Latin America. Evaluation for Transparency ... · evaluations, capacity building...
Evaluation in Latin America. Evaluation for
Transparency, Accountability and Democracy
Gabriela Pérez Yarahuán Claudia MaldonadoTrujillo
Some facts:
It is one of the most unequal
regions in the world (measure by
its Gini Coefficient on income).
The percentage of people living
in poverty has been declining
but is close to a third of the
population and much higher in
some countries.
The mean rate of growth of real
GDP, 2005-2012 for the region
was 3.9. With important variance
among countries.
Most of our nations transitioned
into full electoral democracy
between the mid 90’s and the
early 2000´s.
• By the late 1980´s – 1990´s disruptive political,
economic and social processes that motivated
government change/innovation.
• Accountability, transparency, evaluation
institutions
• New social policy interventions (CCTs)
• Decentralization
• Evaluation for different purposes
–Legitimacy
– Accountability
– Budgeting
– Improvement
Monitoring and Evaluation
Systems
Our concern…
• What do we know about these evaluation
initiatives?
• Are they converging, innovating, providing
good evidence for decision making?
• Are these systems developing evaluation
for responsible innovation?
Studies/Research on M&E Systems
Supply and Demand; Policy Cycle integration; Utilization; and Sustainability
Monitoring and Evaluation
Systems
References Components that characterize the Evaluation Systems
Mackay, K. (2007). How to Build M&E Systems to Support Better Government. Washington D.C.: Banco Mundial.
• Key issues (political, economic or social ) that give rise to the system • Roles and responsibilities of primary stakeholders • Use of the M&E information • Types of monitoring and evaluation • Clear and defined responsibilities for the data collection for the evaluation • Donor support
Cunill-Grau, N., & Ospina, S. M. (2008). Fortalecimiento de los Sistemas de Monitoreo y Evaluación (M&E) en América Latina. Informe Comparativo 12 países. Washington D.C.: Banco Mundial/CLAD.
• Formal existence (legal) • Managing of the system by an entity with authority and functions in the Public
Administration • Steadiness in the M&E activities • Intent global coverage • Use of the information • Explicit articulation of the users and the functions of the system • Placing of the system under the executive branch • Regulatory role • Systematic developed tools and instruments for the execution of monitoring and evaluation
Various studies …
References
Components that characterize the Evaluation Systems
García, R., & García, M. (2010). La gestion para resultados en el desarrollo. Avances y desafios en America Latina y El Caribe. Banco Interamericano de Desarrollo.
Monitoring institutions Scope of programs and projects monitoring Use and dissemination of monitoring information Statistic information systems Legal and institutional framework of the evaluation Actions derived from unattained goals Dissemination of evaluation results
López, G., Mackay, K., & Krause, P. (Eds.). (2012). Building Better Policies: The nuts and bolts of monitoring and evaluation systems. Washington D.C.: Banco Mundial.
• Government demand for information Use of information Availability and quality of government information Officials and consultants skills to evaluate and analyze the system’s
information Available funds for the system
References
Components that characterize the Evaluation Systems
Feinstein, O. (2012). La institucionalización de la evaluación de políticas públicas en América Latina. Presupuesto y Gasto Público, 68(2012), 41-52.
Demand for evaluations o Explicit and periodic requests for evaluations, explicit
stakeholders and engaged funds Supply of evaluation
o Institutions and professionals with experience in hiring evaluations, capacity building activities in evaluation, institutions with potential to hire and perform evaluations
Linkage o Existence of link-up mechanisms and instruments, adjustment
mechanisms between supply and demand, market development of evaluators
Use of evaluations o Use of evaluation results, identification of users, motivations to
use evaluation results
M&E System
Recognition from different stakeholders
Clear planning of monitoring and
evaluation activities, that is transparent
and public
Monitoring and evaluation is sound
and methodologically rigorous
Outputs of the system (monitoring tools and evaluation studies) are used by stakeholders
A developed, institutionalized, successful
system should have the following characteristics:
Recognition from different stakeholders
Clear planning of monitoring and evaluation activities, that is transparent and public
Monitoring and evaluation is sound and methodologically rigorous
Outputs of the system (monitoring tools and evaluation studies) are used by stakeholders
Path towards an adequate system of evaluation
Methodology
• Case studies of 10 Latin American countries–open call
• Qualitative Method of Research: Grounded Theory
• Emphasis in evidence
• Collaborative work with feedback
Country
Argentina
Brazil
Chile
Colombia
Costa Rica
Ecuador
Mexico
Peru
Uruguay
Venezuela
Recognition from stakeholders 2014
Venezuela
Low Argentina
Medium
Brazil, Chile Colombia Costa Rica Ecuador
Mexico, Peru, Uruguay
High
• Legitimate formal rules, norms, regulations for government evaluation functions.
• Rules include relevant government stakeholders to compose a system. • Stakeholders comply, there is evidence that the rules are carried out.
Planning of monitoring and evaluation activities 2014
Ecuador
Venezuela Costa Rica,
Low
Argentina, Brazil Peru,
Uruguay
Medium Chile
Mexico Colombia
High
• Criteria by which programs to be evaluated are selected. • Important decisions which may influence evaluation use • Open access to decision making on evaluation planning. • Explicit evaluation methodologies
Outputs of the system are used by stakeholders
•
•
Use
Conceptual
Symbolic
Instrumental
Argentina, Colombia, Costa
Rica, Ecuador, Nicaragua, Peru,
Dominican Republic, Uruguay,
Venezuela
Brazil, Chile,
Mexico
Low
Medium High
So far…
• There are different models to understand national M&E
systems.
• How to measure progress in M&E systems?
• There is no unique path or recipe of development, but
the common objective is to influence decision making by
providing more and better evidence.
• In order for evidence to influence decision making the
process by which this evidence is collected has to be
institutionalized, this includes the recognition of the
functions of evaluation and M, planning of M and
evaluations, the assurance of methodological rigor and
consistency.
Debates and tradeoffs
Macro institutions
Micro institutions
Another concern…
Institutionalization
Innovation
To conclude…
• M&E systems are developing as a innovation/response to various disruptive
political, economic and social processes…
• In the last 10 to 15 years there have been important developments in the
conformation of M&E systems in Latin America, but this is not true for all
countries.
• Different stages of development
• A need to understand the different aspects of this development and draw
lessons
• Given the recent information we have on M&E systems we can assess
system´s adequacy based on:
– formal and de facto recognition of evaluation functions,
– capacities to develop adequate evaluation planning,
– adequate conduction of evaluations that have methodological consistency and
rigor
– explicit mechanisms to promote different uses of monitoring and evaluations
• Debate on the developments of the
systems based on the best tools to
influence decision making (big institutional
changes vs. micro/demostrations to
promote change).
• Trade off between innovation and
institutionalization?