CGD Policy Paper 52 Perakis Savedoff Does Results Based Aid Change Anything
The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D....
-
Upload
alexander-parsons -
Category
Documents
-
view
216 -
download
1
Transcript of The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D....
![Page 1: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/1.jpg)
The Evaluation GapAn International Initiative to Build Knowledge
Presentation byWilliam D. Savedoff
Sr. Partner, Social InsightProject Director, Center for Global Development
Meeting of the DAC Evaluation NetworkParis
November 10, 2004
![Page 2: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/2.jpg)
The Evaluation GapOverview
• Who are we?
• What do we mean by an evaluation gap?
• Why does this evaluation gap occur?
• What other initiatives are being taken?
• What are we doing?
• Where are we going?
![Page 3: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/3.jpg)
Who are we?
Funding
Project Director
Secretariat
![Page 4: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/4.jpg)
Working Group Members
Members do not represent their institutions but participate in their individual capacity, bringing a wide range of experience from:
• Evaluation Offices• NGOs• Development Agencies• Universities• Philanthropic Foundations
Who are we?Who are we?
![Page 5: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/5.jpg)
Working Group Members
• Nancy Birdsall• Francois Bourguignon• Esther Duflo• Paul Gertler• Judith Gueron• Indrani Gupta• Jean-Pierre Habicht • Dean Jamison• Patience Kuruneri
• Ruth Levine• Richard Manning • Stephen Quick• William D. Savedoff• Raj Shah• Smita Singh• Miguel Szekely• Cesar Victora
Who are we?Who are we?
![Page 6: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/6.jpg)
What is “the evaluation gap”?Range of Analytical Work
• Project preparation studies
• Monitoring implementation
• Process & Operational evaluation
• Outcome evaluation
• Impact evaluation
Who are we?What is the gap?
![Page 7: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/7.jpg)
Impact Evaluation
Studies that measure changes in a target population that can be attributed to a
particular program or policy
Who are we?What is the gap?
![Page 8: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/8.jpg)
Importance of Attribution
0
5
10
15
20
Period
HIV
Pre
va
len
ce
(%
)
• Different studies answer different questions
• Attribution should not be ignored
• Imagine a hypothetical Project for combatting HIV/AIDS in a large country with limited funds …
First RoundBeneficiaryCommunities
![Page 9: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/9.jpg)
Importance of Attribution
0
5
10
15
20
Period
HIV
Pre
va
len
ce
(%
)
• Different studies answer different questions
• Attribution should not be ignored
• Imagine a hypothetical Project for combatting HIV/AIDS in a large country with limited funds …
First RoundBeneficiaryCommunities
Second RoundCommunities
![Page 10: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/10.jpg)
Results from Millions Saved Cases of public health interventions
Excluded• 27 b/c impact could
not be documented• 12 b/c too early or
small scale
Included• 17 identified and
documented
What is the gap?
N
![Page 11: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/11.jpg)
Results from Millions Saved Cases of public health interventions
“The gap in evaluation inhibits the documentation of successes, and prevents policymakers from being able to tell the difference between a well told story and a hard fact as they make decisions about which programs to support.”
What is the gap?
![Page 12: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/12.jpg)
Community Health Insurance
• Community Health Insurance was proposed as early as 1978 in a WHO Technical Expert Report
• What have we learned about community health insurance in the 36 years that have followed?
What is the gap?
![Page 13: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/13.jpg)
Community Health InsuranceILO/Universitas Review
What is the gap?
127 Studies of 258 Cases
24 Cases Measured Utilization
Two Cases Had Internal Validity
None of the studies measured impact on health
![Page 14: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/14.jpg)
Community Health InsuranceEkman, Health Policy & Planning (2004)
What is the gap?
Descriptive Non-Statistical
(22)
Other (2)
Multivariatestatistical (5)
Descriptive (14)Review of studies that analyzed:1. Resource Mobilization2. Financial Protection
![Page 15: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/15.jpg)
Why does this gap occur?
• Knowledge from impact evaluations is a public good
• Costs are visible, benefits seem far-off & resources are limited
• Demand is diffuse (institutions & time)• “It Pays to be Ignorant” • Low-quality studies crowd out the good• Methodological challenges
Why?
![Page 16: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/16.jpg)
Guarded optimism
• Changing profile of agency staff• Worldwide capacity for good studies• Recognition of impact evaluation value• Pressure from skeptical donors • Methodological & practical advances in research
design (also highly visible & successful)– PROGRESA/Oportunidades, IMCI, Guinea Worm,
etc.
Why?
![Page 17: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/17.jpg)
Other Initiatives
• Increasing access to existing information
• Developing aggregate indicators
• Improving capacity
• Promoting evaluation w/funds & data
• Conducting research & demonstrating good evaluation practices
Other Initiatives
![Page 18: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/18.jpg)
Some Examples
• OECD/DAC Evaluation Network
• World Bank Research Department
• Health Metrics Network
• UN M.E.R.G.
• USAID/MACRO Surveys
• MIT Poverty Action Lab
Other Initiatives
![Page 19: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/19.jpg)
We need an initiative that:
• Focuses specifically on the Public Good aspect of impact evaluation
• Develops a collective response to the problem
• Mobilizes & appropriately channels new funds
• Acts selectively where the most can be learned
![Page 20: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/20.jpg)
The Evaluation GapWorking Group Process
• Preliminary interviews & research
• Convene working group
• Meetings, teleconference, and e-list debates
• Consultation group
• Draft paper & action plan
• Dissemination and broader debate
What are we doing?
![Page 21: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/21.jpg)
Likely characteristics of recommendations
• Identifying a reliable source of funding
• Establishing collective mechanism for selecting “enduring questions”, guidance, involvement & commitment
• Developing institutional mechanisms for channeling funds into appropriate studies & projects
Where are we going?
![Page 22: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/22.jpg)
What do you think?
• What is the fundamental problem from your perspective?
• Do you have other examples (reviews?)
• What are your current initiatives?
• What kinds of solutions should be considered?
• Who else should we consult?
Where are we going?
![Page 23: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/23.jpg)
Contact Us
Center for Global Developmentwww.cgdev.org
• Nancy Birdsall, [email protected]
• Ruth Levine, Senior [email protected]
• William D. Savedoff, Project [email protected]
Where are we going?
![Page 24: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/24.jpg)
References
• Christensen, Jon. Asking the Do-Gooders to Prove They Do Good. The New York Times . 1-3-2004.
• Development Assistance Committee. Principles for Evaluation of Development Assistance. Organisation for Economic Co-operation and Development. 1991. Paris, OECD. 2004.
• Development Assistance Committee. Review of the DAC Principles for Evaluation of Development Assistance. Organisation for Economic Co-operation and Development. 1-120. 1998. Paris, OECD. 2004.
• Development Assistance Committee. Glossary of Key Terms in Evaluation and Results Based Management. 6, 1-37. 2002. Paris, OECD. Evaluation and Aid Effectiveness.
![Page 25: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/25.jpg)
References
• Dugger, Celia. “World Bank Challenged: Are the Poor Really Helped?” The New York Times . 7-28-2004. New York.
• Ekman, Björn. "Community-based health insurance in low-income countries: a systematic review of the evidence." Health Policy and Planning, 2004, 19 (5), 249-270.
• France, Ministère de l'Economie des Finances et de l'Industrie. Partners in Development Evaluation: Learning and Accountability. Partners in Development Evaluation: Learning and Accountability. 3-25-2003. Paris.
![Page 26: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/26.jpg)
References
• International Labour Office. 2002. “Extending Social Protection in Health Through Community Based Health Organizations: Evidence and Challenges”, Dicussion Paper, Universitas Programme, ILO, Geneva.
• Kremer, Michael. "Randomized Evaluations of Educational Programs in Developing Countries: Some Lessons." American Economic Review Papers and Proceedings, 2003, 93 (2), 102-115.
• Pritchett, Lant. "It Pays to Be Ignorant: A Simple Political Economy of Rigorous Program Evaluation." The Journal of Policy Reform, December 2002, 5 (4), 251-269.
![Page 27: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/27.jpg)
References
• Victora, Cesar G, Habicht, Jean-Pierre, Bryce, Jennifer. "Evidence-Based Public Health: Moving Beyond Randomized Trials." American Journal of Public Health, March 2004, 94 (3), 400-405.
• World Bank. Influential Evaluations: Evaluations that Improved Performance and Impacts of Development Programs. 1-24. 2004. Washington, DC, World Bank.
• World Health Organization. Financing of Health Services. 625. 1978. Geneva. Technical Report Series.
![Page 28: The Evaluation Gap An International Initiative to Build Knowledge Presentation by William D. Savedoff Sr. Partner, Social Insight Project Director, Center.](https://reader036.fdocuments.us/reader036/viewer/2022070412/56649e4f5503460f94b47359/html5/thumbnails/28.jpg)
EG Working Group Members• Nancy Birdsall, President, Center for Global Development• Francois Bourguignon, Chief Economist & Sr. Vice President, World Bank• Esther Duflo, Associate Professor of Economics, MIT • Paul Gertler, Professor of Economics, Haas School of Business • Judith Gueron, President, MDRC• Indrani Gupta, Reader, Institute of Economic Growth• Jean-Pierre Habicht, Professor, Cornell University • Dean Jamison, Senior Fellow, National Institutes of Health• Patience Kuruneri, Senior Policy Analyst, World Health Organization• Ruth Levine, Senior Fellow, Center for Global Development• Richard Manning, Chair, Development Assistance Committee • Stephen Quick, Director, Inter-American Development Bank• William D. Savedoff, Senior Partner, Social Insight• Raj Shah, Senior Policy Officer & Senior Economist, Bill & Melinda Gates Foundation• Smita Singh, Special Advisor for Global Affairs, William & Flora Hewlett Foundation• Miguel Szekely, Ministry of Social Development, Mexico• Cesar Victora, Professor, Universidade Federal de Pelotas, Brazil