World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael...
-
Upload
leonard-hall -
Category
Documents
-
view
214 -
download
0
Transcript of World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael...
![Page 1: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/1.jpg)
World BankIndependent Evaluation Group
How to Increase the Utilization of Evaluations
Michael Bamberger
![Page 2: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/2.jpg)
2
Session outline
1. Defining and measuring evaluation utilization
2. Reasons why many evaluations are under-utilized
3. Examples of evaluation utilization: the World Bank “Influential Evaluations” study
4. Ways to strengthen utilization5. If time permits .. Further discussion on
“presenting the message”
![Page 3: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/3.jpg)
1. Defining and
measuring
evaluation utilization
![Page 4: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/4.jpg)
4
Defining evaluation outcomes and impacts Use
• How evaluation findings are utilized by policymakers, managers and others
Influence• How the evaluation influenced decisions and actions
Consequences• How the process of conducting the evaluation, the findings
and the recommendations affected the agencies involved and the target populations..
• Consequences can be:• Positive or negative
• Expected or unanticipated
![Page 5: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/5.jpg)
5
Measuring evaluation outcomes and impacts Changes in individual:
• Knowledge• Attitudes • Behavior
Changes in organizational behavior Changes in program design or
implementation Changes in policies and planning Decisions on project continuation, expansion
and funding.
![Page 6: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/6.jpg)
6
Measurement issues
Time horizon Intensity Reporting bias
• Many agencies do not acknowledge they have been influenced
Attribution: • How do we know the observed changes were
due to the evaluation and not to other unrelated factors?
![Page 7: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/7.jpg)
7
Attribution analysis
How do we know if observed changes were due to the evaluation and not to other influences?
Stakeholder interviews Surveys [pretest/posttest or posttest only] Analysis of planning and policy documents for
evidence of evaluation influence Analysis of mass media Key informants
![Page 8: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/8.jpg)
8
Examples of attribution methodology used in “Influential Evaluations”
See:
“Influential evaluations: Detailed case studies” pp. 69-72
![Page 9: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/9.jpg)
Attribution analysis framework
1. Identify potential Impacts (effects)
2. Assess whether there is a plausible case forattributing part of the
effects to the evaluation
3. What proportion of the effects can be attributed
to the evaluation?Triangulation
![Page 10: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/10.jpg)
10
A. Comparison of 2 user surveys + Stakeholder opinion survey
Comparison of sample surveys of service users (in 1993 and 1999) found reported improvement of services [potential impact]
Sample of 35 public service agencies, policy-makers, mass media and civil society [corroborated influence of survey in influencing improvements].
Bangalore Citizens Report Cards
![Page 11: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/11.jpg)
11
B. Testimonials from stakeholders
Clients (Metallurgical company, Development Bank; AUSAID Project Dept, Indonesian water agency) asked to send testimonials in letter or e-mail confirming influence of the evaluation.
In Bulgaria clients asked to confirm validity of benefit projections and to confirm benefits were attributable to the evaluation.
Bulgaria: Metallurgical Project and
Indonesia: Village Water Supply
![Page 12: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/12.jpg)
12
Considered too bureaucratically difficult to solicit Government views on effectiveness of government agency (Evaluation Organization).
Opinions of Bank Resident Mission and other experts solicited on the influence of the evaluation and the credibility of the estimates of cost-savings and employment generation.
Paper trail: specific references in the Five Year Plan and follow-up sector planning documents to how the evaluation was used.
C. EXPERT ASSESSMENT + PAPER TRAIL
India: Employment Assurance Program
![Page 13: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/13.jpg)
13
D. Logical deduction from secondary sources
Follow-up PETS study estimated increased funds utilization (potential evaluation impact)
Extensive coverage of the report in the media Government documents show how the findings
were used Reports show how community groups use the
budget information posted in schools/ media.
Public expenditure tracking study (education): Uganda
![Page 14: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/14.jpg)
2. Reasons why evaluations are under-utilized
![Page 15: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/15.jpg)
15
# 1 Lack of ownership
Evaluation focus and design are determined by donor agencies or outside “experts” with little real input from client.
The “goals definition game” alienates clients
Limited consultation with, and feedback to, clients.
Evaluation seen as a threat
![Page 16: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/16.jpg)
16
# 2 Timing
The evaluation findings are • presented too late to be useful
• Too soon – before policymakers or managers have started to focus on the issues discussed in the report
![Page 17: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/17.jpg)
17
# 3 Poor communication between evaluator and client
Clients are not kept in the loop Clients may not like evaluator’s communication
style Language problems Conceptual problems The “objectivity” paradigm limits contact and
communication between evaluator and client Client does not share information with other
stakeholders
![Page 18: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/18.jpg)
18
# 4 Lack of flexibility and responsiveness to client needs
Rigid design that cannot be adapted to client needs or changing circumstances
Quasi-experimental design that cannot adapt indicators and data collection methods to changing circumstances.
“Objective” stance of evaluator limits interaction with clients.
Timing: too early or too late[continues next page]
![Page 19: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/19.jpg)
19
Finance ministries try to force evaluation indicators and focus to correspond to budget line items
National evaluation systems sometimes introduce top-down, uniform evaluation/reporting systems not reflecting the reality of different agencies
![Page 20: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/20.jpg)
20
# 5 Resource constraints
Budget constraints affect• Data collection
• Data analysis
• Bringing staff together to participate in the evaluation process
• Translation into local languages
• Report preparation and dissemination
Limited local expertise
![Page 21: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/21.jpg)
21
# 6 Time constraints
Too many demands on client and stakeholders’ time
The evaluators do not have enough time to:• Consult with clients during evaluation planning
• Design and implement the evaluation properly
• Discuss the draft report with clients
• Organize effective dissemination meetings
![Page 22: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/22.jpg)
22
# 7 Relevance
The evaluation does not address priority information needs of clients
Much of the information is not considered useful
The information is not analyzed and presented in the way that clients want:• Too detailed
• Too general
![Page 23: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/23.jpg)
23
# Factors external to the evaluation affecting utilization
Problems with the evaluation system Dissemination mechanisms Political ethics (attitudes to transparency) Client’s lack of long-term vision Government perception of evaluation
![Page 24: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/24.jpg)
3. Examples of evaluation utilization
The World Bank “Influential
Evaluations” study
![Page 25: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/25.jpg)
25
How are evaluations used? When are they influential?
1. Evaluation never the only factor. How does the evaluation complement other sources of information and advice.
2. Political cover for difficult decisions3. Identifying “winners” and “losers” and
showing how negative impacts can be mitigated.
4. Credibility and perceived independence of the evaluator may be critical
[continued next page]
![Page 26: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/26.jpg)
26
5. The big picture: helping decision-makers understand the influence of the social, economic and political context.
6. Help managers understand how political and other pressures limit project access to certain groups
7. Providing new knowledge or understanding 8. Catalytic function: bringing people together or
forcing action.
![Page 27: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/27.jpg)
27
Types of influence that evaluations can have
1. India: Employment Assurance• Broader interagency perspective helped identify
duplications and potential cost savings.
• Evaluation Office had high-level access to Planning Commission
2. India: Citizen Report Cards• Alerting management to service problems and
• providing quantitative data to civil society pressure groups
3. Indonesia: Village Water Supply• Making policy-makers aware of importance of gender issues
and participatory approaches
[continued next slide]
![Page 28: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/28.jpg)
28
4. Large Dams• Created political space for introducing new social
and environmental criteria for evaluating dams and
• launching dialogue that facilitated creation of World Commission on Dams.
5. Pakistan: Wheat Flour Ration Shops• Political cover for sensitive political decision
• Showed how to mitigate negative consequences
[continued next page]
![Page 29: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/29.jpg)
29
6. Uganda: Education expenditures• Developed methodology to document what
everyone suspected (expenditure wastage) • provided documentation to civil society to
pressure for improvements
7. Bulgaria: Metallurgical Project• Alerting borrowers and Development Bank to new
EU legislation• showing how to avoid fines • how to advance launch of mineral production
[continued next page]
![Page 30: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/30.jpg)
30
8. China: Forestry Policy• Legitimized questioning the logging ban
• promoting more in-depth policy research
• facilitating creation of Forestry Task Force
![Page 31: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/31.jpg)
31
1. Major cost savings (India, Bulgaria, Pakistan)
2. Increased financial benefits (Uganda, Bulgaria)
3. Forced action (Bangalore, Uganda)
4. Strengthened gender and participatory planning and management of water (Indonesia)
5. Introduced social assessment of dams but discouraged future investments (Dams)
[Continued next slide]
What difference did the evaluation make?
![Page 32: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/32.jpg)
32
6. Increased efficiency of service delivery (India, Bangalore, Indonesia)
7. Facilitated creation of important policy agency (Dams, China)
![Page 33: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/33.jpg)
4. Ways to strengthen evaluation utilization
![Page 34: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/34.jpg)
34
Ways to strengthen evaluation utilization
# 1. Deciding what to evaluate# 2. Timing:
• When to start• When to present the findings
#3. Deciding how to evaluate• Choosing the right methodology
#4. Ensuring effective buy-in• Stakeholder analysis and building alliances• The importance of the scoping phase• Formative evaluation strategies• Constant communication with clients
![Page 35: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/35.jpg)
35
#5. Evaluation capacity building
#6. Deciding what to say [see next section]
#7. Deciding how to say it [see following section]• Effective communication strategies
#8. Developing a follow-up action plan
![Page 36: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/36.jpg)
36
# 6. Deciding what to say
Technical level Amount of detail Focus on a few key messages Target messages to key audiences
![Page 37: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/37.jpg)
37
Sources of lessons about a program
Evaluation findings
Experience of practitioners
Feedback from program participants
Expert opinion
Cross-discipline connections and patterns
Strength of linkages to outcomes
![Page 38: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/38.jpg)
38
Identifying evaluation lessons and generating meaning
Tactics for generating meaning (Handout 1)
Identifying high quality lessons (Handout 2)
![Page 39: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/39.jpg)
39
# 7. Presenting the message
1. Communication style and choice of media (Handout 3)2. Focus report on intended users3. Quantitative and qualitative communication styles
(Handout 4)4. The clients preferred communication style (Handout 5`)5. Making claims6. The importance of graphics7. Who receives the evaluation report and who is invited to
comment
![Page 40: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/40.jpg)
40
If time permits ……..
More detailed discussion on presenting the message
![Page 41: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/41.jpg)
41
Presenting the message
1. Communication style and choice of media2. Utilization-focused reporting principles3. Quantitative and qualitative communication
styles4. The client’s preferred communication style5. Rules for written reports6. Making claims7. The importance of graphics8. Who receives the report and who is invited to
comment
![Page 42: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/42.jpg)
42
1. Communication style and choice of media
Continuous communication throughout the evaluation1
• “No surprises”
• Educating the client how to think about evaluation
Short versus long reports Combining verbal and written presentations Slide shows Informal versus formal
![Page 43: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/43.jpg)
43
Communication style .. continued
Alternative media• Internet
• Video
• Theater
• Dance
• Murals/paintings
• Posters
• Signs
![Page 44: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/44.jpg)
44
Communication style .. continued
Using the right language• Making sure the written report is available in
stakeholder languages
• Economical ways to translate
Personal testimony Project visits Working with the mass media
![Page 45: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/45.jpg)
45
2. Utilization-focused reporting principles
Be intentional and purposeful about reporting
Focus reports on primary intended users Avoid surprising stakeholders Think positive about negatives Distinguish dissemination from use
Source: Patton (1997): 330-337
![Page 46: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/46.jpg)
46
3. Quantitative and Qualitative communication styles
Tables versus text Types of evidence
• Statistical analysis
• Case studies
• Stories
• Photos
• Boxes
• Site visits
• Testimonials
![Page 47: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/47.jpg)
47
3. The client’s preferred communication style
Written and/or verbal Quantitative/qualitative Multiple presentations to different
audiences
![Page 48: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/48.jpg)
48
Understanding the communication style of the decisionmaker
High intellectual level: Robert MacNamara, Elliot Richardson
“These two individuals were perfectly capable of understanding the most complex issues and absorbing details – absorbing the complexity, fully considering it in their own minds.”
Lawrence Lynn. Prof. Public Administration. The Kennedy School. Patton 1997. PP 58-9.
![Page 49: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/49.jpg)
49
Decisonmakers communication styles .. continued
Short personal stories: Ronald Reagan
Preferred Reader’s Digest type personal stories and anecdotes.
![Page 50: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/50.jpg)
50
Decisonmakers communication styles .. continued
Political animals: Joseph Califano [U.S. Secy Health, Education and Welfare
“Califano is a political animal and has a relatively short attention span – highly intelligent, but an action-oriented person”. The problem that his political advisers had is that they tried to educate him in the classical rational way, without reference to any political priorities .. or presenting alternatives that would appeal to a political, action-oriented individual.”
Source: Patton (1997) p.59.
![Page 51: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/51.jpg)
51
5. Rules for written reports
Show, don’t tell Be brief and clear Build the story with paragraphs Write clear sentences Omit needless words Avoid jargon
[Vaughan and Buss]
![Page 52: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/52.jpg)
52
Reporting results
Arranging data for ease of interpretation• Focusing the analysis
Simplicity in data presentation Interpretation and judgment Making claims Useful recommendations A futures perspective on recommendations
Source: Patton (1997): pp 321-4
See next slide
![Page 53: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/53.jpg)
53
5. Making claims
Importance of claims
Major Minor
Rigor of claims
Strong
Weak
![Page 54: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/54.jpg)
54
Characteristics of claims of major importance
Involves having an impact Deals with important social problem Affects large numbers of people Saves money and/or time Enhances quality Show something can really be done about a
problem Involves model or approach that could be
replicated
![Page 55: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/55.jpg)
55
Characteristics of strong claims
Valid, believable evidence Data covers a long period of time Claim is about clear intervention with solid
documentation About clearly specified outcomes and impacts Includes comparisons to program goals, over
time, with other groups, with general trends or norms
![Page 56: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/56.jpg)
56
Strong claims … continued
Evidence for claims includes replication• More than one site
• More than one staff member obtains outcomes
• Same results from different cohort groups
• Different programs obtained comparable results using the approach
Based on more than one kind of evidence Clear, logical linkages between intervention
and claimed outcomes Evaluators independent of staff
![Page 57: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/57.jpg)
57
6. The importance of graphics
Line graphs: trends over time Pie charts: parts of a whole Cluster bar chart: comparing several
items Combination chart: bar chart and trend
line *** Don’t overload graphs ***
![Page 58: World Bank Independent Evaluation Group How to Increase the Utilization of Evaluations Michael Bamberger.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649f475503460f94c69317/html5/thumbnails/58.jpg)
58
7. Who receives the evaluation report and who is invited to comment?
Restricted to few key decision-makers and managers
Participatory presentation/ consultation with stakeholders
Public comment• Public hearings
• Via civil society Copies available to the press