Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

16
Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation November 14, 2009 Dale Hill Independent Evaluaton Group, World Bank

description

Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation. November 14, 2009 Dale Hill Independent Evaluaton Group, World Bank. Key Messages. 1st evaluation likely to be easier & more successful if “enabling conditions” are in place early - PowerPoint PPT Presentation

Transcript of Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Page 1: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Good Practice Planning Tips for GRPP Decision-Makers…at time of EvaluationNovember 14, 2009Dale HillIndependent Evaluaton Group, World Bank

Page 2: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Key Messages► 1st evaluation likely to be easier & more successful if

“enabling conditions” are in place early • If not in place, situational analysis essential before planning• Can then build in ways to compensate & progress on

conditions► When assessing timing & readiness for evaluation, need

to plan well to assure independence & needed expertise • More important for GRPPs – both needed at all stages• For independence, Governing Body makes most key decisions

► Early planning also needs to consider audience, Stakeholder involvement and dissemination needs.

2

Page 3: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Special Features of GRPPs in Relation to Evaluation Planning► High expectations of evaluation – to guide

development aid allocation decisions ($b) and yield broad lessons

► Open-ended programs with multi-stakeholder governance; scope and design evolve over time

► Governance & management structure raise issues of organizational independence & adequate expertise

► Broad objectives & reach; often complex multi-level operations – global, regional, country, local

► Management faces challenges of navigating complex decision-making to respond to Governing Body

3

Page 4: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Revisiting Enabling Conditions

► “Intentionality” – Formal Commitment to M&E and to act on findings

► Roles for M&E Defined in Advance► Formal M&E Policy and/or Principles► Results Framework

•Agreed Objectives at Founding•Monitoring Framework/logical framework• Follow through – Design System +

Baseline Data► Examples Beyond GEF at end of slides for

takeaway4

Page 5: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Good Practice (GP): Enabling Conditions

► Early Provision for framework, baseline data collection and monitoring

► Expert Panel set up w/Evaluation role early; Roles Defined

► UNAIDS (several evaluations completed)

► Millennium Villages (early evaluation completed only)

► GFATM (1st program-wide evaluation completed)

► GAVI (1st evaluation completed; 2nd underway)

5

Conditions Set up for First Evaluation (Rare Cases)

Page 6: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

1st Step: Situational Analysis► Because of high expectations, planning doubly

important ► Before or soon after call for evaluation, Mgmt.

should do situational analysis & place evaluation on GB agenda

Management should ask: ► What is driving demand for evaluation at this time?► How should we take account of or build on past

evaluations (if applicable)?► What is our readiness for an evaluation at this time?► If not ready, what options do we have?

6

Page 7: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Timing and Past Evaluations….High Expectations, & Multiple Stakeholders► What is Driving Timing? Who asked for evaluation?

• Single donor may want for accountability – w/deadlines set• But 1st best is evaluation commissioned by program – with

FULL Governing Body consensus on need & timing► How to take account of past evaluations?

• Take account of whether the past evaluations met all expectations – may want better quality, coverage this time

• Δ in global context may dictate different approach, coverage• Multiple single-donor evals of GRPPs common – avoid

duplication and consider program-level meta-evaluation

7

Page 8: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Readiness Assessment► Status of consensus on need, timing & approach

• Objectives of program and log frame up to date? • New constituencies to involve and include?

► Status/Access to funding• 1st best – program budget for evaluation–but if not pre-

planned –• Can get GB approval of special item – build in time for this• 2nd best – if donor(s) offer to fund, prevent undue influence

► Status of GB & staff expertise on evaluation? Access to support?

► Status of info: (Usual) Baseline/monitoring, program records• Sometimes overlooked – enough activities completed?

► Consider audiences, translation, dissemination needs

8

Page 9: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Consequences of Poor PlanningPoor Planning: w/no compensation for Constraints

Constraints► Not enough activities

completed► Inadequate budget for

evaluation, dissemination

► Monitoring system not in place

► Not taking into account previous evaluations

Consequences► Low yield; limited findings

► Poor quality; compromised credibility; less used, accessible

► Eval. Q’s less clear; new data collection $$; findings limited

► Costly to program; burden on informants; may confuse strategy

Page 10: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Setting Broad Purpose & Design

► Should flow from Situational Analysis• e.g. Accountability for donor; Assess Impact of

Δ’d Context► Ideally Approved by Governing Body► Usually varies by Program Stage or

Maturity

10

New, Young Fine-tune design; confirm results chain; identify main risks; display results to satisfy donors

Evolved, Steady-State

Fill Accountability needs; Δ governance, strategy as needed; broaden lessons& influence

Evolved, but Δ’d context

Assess implications of changed context for strategy, objectives, reach and coverage

Mature Assess likelihood of sustainability of outcomes;Consider devolution

Page 11: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Assigning Roles: Key Players in the Evaluation► Governing Body► Overseers of Evaluation

• Oversight Committee• Standing Technical Expert Panel – predefined roles or

not► Program manager and staff► Independent Evaluation Team (usually consultants)► Others: Keep in mind for consultation, dissemination

• Key partners not represented on GB (donors)• Other Key informants (e.g. past GB members…)• Implementing Agencies (need translation?)• Direct Beneficiaries (need different, briefer

product?)

11

Page 12: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Preferred Roles of GB & Management

12

GB/Oversight Committee ManagementApproves M&E policy Upstream: May draft M&E policyConsiders timing & purpose;Commissions evaluation, Approves roles

Calls for or receives call for evaluation Does situational analysis, proposes Δ’d timing, budget, approach & roles

Approves criteria & process for selection

Proposes criteria &process for team selection

Approves selection Manages selection process; seeks expert input

Oversees evaluation Briefs team, provides documentation, Provides day-to-day admin. support

Receives first draft; may share w/Mgmt

Receives first draft in parallel w/or after GB

Approves final draft Free to comment & correct factual errors

May draft separate GB response

May draft Mgmt response

Approves responsive action Elaborates responsive action

Page 13: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Roles of Others: Participation & Consultation Upstream & Downstream

► Principles/mandate in charter, M&E Policy or legal agreements?• If not, Mgmt should consult GB and determine expectations

& Consider:• Before (design), During (interviews/surveys), After

(review/inform findings)► If no Formal Stakeholder List or Map – Prepare, to

equip evaluators

13

1st best – DESIGN – Global Alliance for Vaccines and Immunization (GAVI). The program decided to hire consultants to conduct a pre-evaluation “scoping Study” to survey a wide range of stakeholders to submit evaluation questions relevant to them. A Steering Committee provided a proposed list of stakeholders to the consultants. The submitted evaluation questions were compiled, consolidated and then sent back to stakeholders for prioritization. The resulting list of priority questions was referenced in the subsequent evaluation TOR/RFP as forming the basis for the evaluation.

2nd best – DURING – Evaluators of the One World Trust worked with the program to develop a matrix with categories of stakeholders in order to define a suitable sampling frame for surveys for the evaluation.

Page 14: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Planning: Design, Budget, Contracting

...Expertise Needed on Evaluation

...Weighing all Options and Responding to GB imperatives► Design must respond to purpose & match complexity of program

plus budget—credibility essential ► If expertise available (Oversight committee, host agency) – can

have confidence in design, budget adequacy and feasibility of work program• TOR can be specific; team selection mainly on qualifications

► If not, several options:• 2-stage evaluation: “evaluability study” or experts draft TOR;• Leave details to evaluators & selection process

– RFP has proponents submit design/work program (sometimes budget)

– Inception report/early consultation on methodology desirable

14

Page 15: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Conclusions & Hopes for the Future

► Ideally, the good practices we highlight will lead to more frequent & earlier attention to the enabling conditions which set the stage for successful GRPP evaluations.

► Where that is not possible, we hope decision-makers will see the benefits of doing a situational analysis to inform early planning and ensure a realistic, consensus approach.

► We further hope that such early planning will increasingly:• Lead to assigned roles in evaluation which ensure organizational

independence & access to expertise at key stages; • Adequately address questions of participation and

dissemination;• And thus lead to higher quality evaluations & better

development.

15

Page 16: Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

Thank You