Designing a Better DPF Results Framework

October 2, 2018

DPF contributes to countries’ results by supportingpolicy and institutional reforms

Program Development

Objective

Prior Actions

Results

The Program Dev. Objective is selected from the country’s own development program/strategy

World Bank supports key policy/ institutional actions, drawn from country program

Results from prior actions are used to monitor and evaluate impact. Results are drawn from government programs.

2

Prior Actions

Prior Actions

• An action changes something: it changes incentives, or prices, or governmental structures, etc.

• Policy actions frequently take the form of new laws or regulations or decision rules (e.g. a new tariff-setting mechanism)

• Institutional actions – focus on function, rather than organizational form

• Statements of policy, reports and studies are not usually policy actions. They might list intended future actions, but do not themselves change anything

• The complexity of the program content (prior actions and triggers) should be calibrated to the government’s capacity and “ownership”

3

Results

Results

• Specific, not vague

• Caused by the prior actions and triggers, viz. not so distant as to raise questions about attribution

• Different from the prior action that caused it. It should not be a rewording of the prior action

• Outputs or outcomes? preference for outcomes – more likely to be able to demonstrate that the PDO has been reached. Sometimes difficult to specify outcomes, in which case outputs are the only alternative

• Quantitative, not qualitative, whenever possible

• Time-bound – with a baseline value and a target value, both of which have a date. These should be cited at the outset

• Focused on the end of the program

4

Linking program development objectives with results

Program Development

Objective

• Key questions: What does the government want to accomplish? How will we know it is achieved?

• Each development objective must be measured by one or more results in the results framework

• Failing this it is impossible for the task team to evaluate whether the objectives have been attained

• “Daylight” between PDOs and results may give a negative evaluation of the program

5

Systems for monitoring and evaluation

• Monitoring and evaluation are the responsibility of the client

• The program document assesses the client’s implementation and M&E capacity and M&E systems

• Describe M&E arrangements and capacity enhancement measures (if any) in program document

• For a programmatic series, the ICR report is prepared at the end of the series

• The Bank’s own evaluation process is focused on results – both the ICR report and the IEG review of it

6

Indonesia: using DPF to address critical constraints

Reforming the Indonesian maritime logistics sector ($400 m, FY17, P158140). To reduce costs of logistics, by privatization of ports and eliminating importer licenses.

Sustainable and Inclusive Energy DPF ($500 m, FY16, P154291). To support reforms that: (a) reduce the fiscal cost of electricity; and (b) expand access to modern energy.

Indonesia Fiscal Reform DPL 2 ($300 m, FY18, P161475). To support fiscal reforms that: broaden the tax base, improve revenues, and make public spending more efficient.

7

Indonesia maritime logistics – an institutional reform or a transaction?

8

Prior Action: Port Authorities grant concessions to operate 107 ports to Pelindos (Indonesia Port Corp.)

Result: Increase in the number of Pelindos’ ports with internationally certified management systems. Baseline (2014): 46, target (2018): 60

PDO: Improving port efficiency and governance

Indonesia Energy – is an approach paper more than just a statement of intent?

9

Prior Action: Issue an electrification approach with improved coordination of institutional responsibilities and financing mechanisms.

Result: Increase in household electrification rate:Baseline (2014): 84.35 percent Target (2017): 92.75 percent

PDO: Expanding access to modern, reliable energy

Indonesia Fiscal Reform – why budget allocations are ad hoc reforms

10

Prior Action: Raise the share of the budget allocated to infrastructure, social assistance and health sectors.

Result: Budget allocations rise from baseline in 2014 to target in 2018: social assistance: 2.7% 4.5%; health: 4.0% 7.4%; infrastructure: 12.3 19.5%

PDO: Improved composition of spending

Plans as policy actions?

11

• Can plans, strategies, roadmaps, statements of

policy, policy frameworks, etc. be used as prior

actions or triggers?

• YES, in programmatic series of DPF, if followed up

with implementation leading to results

• NO, in stand-alone DPF and the last operation of a

programmatic series, because:

➢ Plans on their own have few or no results

➢ May indicate a lack of understanding of the problem

➢ May signal insufficient preparation to have an impact

➢ May indicate a lack of commitment

A credible plan: a decision rule with follow-up implementation

12

Prior action, 2018: Council of Ministers approves a multi-year electricity tariff framework.Trigger, 2019: Carry out reform measures for a second year under the tariff framework.Trigger, 2019: Carry out reform measures for a third year under the tariff framework.

Result: Tariff rate rises from 0.57 Birr/kWh (2017) to 1.95 Birr/kWh (2021)

?

A credible plan: a decision rule with follow-up implementation. But: back-loading & risk.

13

Prior action, 2018: Macroeconomic Committee endorses a white paper on telecom sector reform permitting competition.Trigger, 2019: Independent regulator created.Trigger, 2019: Advisers appointed for issuance of additional telecom licenses.

Result: Number of telecom licensees operating rises from 1 (2018) to 3 (2021).

?

A doomed plan: No decision rule, no follow-up implementation

14

Prior action, 2018: Council of Ministers has approved Guidelines for privatizing SOEs.Trigger, 2019: Submit to Parliament an SOE Proclamation about state ownership & governance.Trigger, 2019: Regulation authorizing ownership entities such as holding companies.

Result: Number of SOEs that publish their audited financial statements rises from 1 (2018) to 9 (2019)

?

An ‘orphan’ PDO – where is the result?

15

Results: (i) % of districts with labor-intensive public works coverage rises from X to Y. (ii) # of beneficiaries of social protection payments rises from X to Y. (iii) # of social protection staff rises from X to Y.

PDO: To support the government to enhance efficiency and expand coverage of social protection

?

Budget allocations as policy actions?

16

Not preferred:▪ Contrary to the principle of the unity of the

budget – because ‘earmarking’▪ Intrusive ▪ May indicate a lack of commitment, in which

case the results chain is jeopardized▪ An illustration of this in a recent operation

follows

A budget allocation that failed due to lack of commitment

17

Trigger for 2018, listed in 1st operation in 2017: Funding for performance based financing is budgeted. Prior action for 2018, in 2nd op., concept, 2018: FCFA 2.25 billion included in Amended Finance Law 2018.Prior action for 2018, in 2nd op., ROC, 2018: Blank.

Result: Share of facilities offering essential health services by qualified health workers rises from 11% (2017) to 40% (2020).

?

An ‘empty nester’ trigger with no result

18

Prior action, 2018: Decree to deploy health personnel in underserved areas. Trigger, 2019: Apply Workload Indicators of Staffing Need (WISN). Trigger, 2020: Online platform to ensure greater transparency in human resources management.

Result: Fraction of rural clinics that meet WISN standards rises from 20% (2018) to 60% (2021).

PDO: Improve health care service delivery

19

Prior action, 2014: Draft law to Parliament revoking possible use of land as bank capital (thus removing a potential distortion and governance risk in the landmarket and banking system). Prior action, 2015: Said law enacted (after being passed).

Result: Blank.

PDO: Strengthen the regulatory framework and reduce the costs of doing business.

‘Empty nester’ prior actions with no result

20

Prior actions, 2014 & 15: Ease business & property registration, reduce # of permits, harmonize food safety standards with EU, protect rights of investors.

Result: Reduce business permits from 143 (in 2013) to 84 (2017).

PDO: Strengthen the regulatory framework and reduce the costs of doing business.

‘Empty nester’ actions with no results – can we aggregate?

More ‘empty nester’ actions with no results

21

Prior actions, 2014 & 15: Raise gas prices; make energy regulator independent; restructure the gas sector; modify incentives so as to attract investment to the sector; restructure the gas utility and distribution companies; install gas and heat meters.

Result: Financial deficit of the energy utility falls from $X millions (in 2013) to $Y millions (2016).

PDO: Reform inefficient and inequitable utility subsidies while protecting the poor.

A poorly timed results chain does not give enough time for an ambitious result

22

Prior action, 2018: Update norms for the construction and equipment of medicine depots. Trigger, 2019: Adopt an integrated national information system for logistics management of the supply chain for medications. Trigger, 2020: Implement said system.

Result: Fraction of health facilities that are short of medications falls from 86% (2017) to 20% (2021).

PDO: Improve health care service delivery

Good sequencing, but what is the policy that ‘makes something operational’?

23

Prior action, 2018: Make operational the National Fund for Universal Health Insurance. Trigger, 2019:Assess the cost efficiency of targeted free health care programs, and adopt corrective measures. Trigger, 2020: Adopt a mechanism for coverage of the informal sector by the NFUHI.

Result: Fraction of population covered by universal health insurance rises from 0% (2017) to 32% (2021).

PDO: Improve health care service delivery

A result measure that is too distant from the prior actions to be meaningful

24

Prior action, 2017: Rationalize tax expenditures of the investment code; harmonize VAT exemptions with WAEMU norms. Prior actions, 2018: Deploy ASYCUDA at 8 border posts; digitize payment at customs and tax offices; create 11 new tax offices and 1 property tax center.

Result: Tax revenue rises from 13.6% of GDP (2016) to 14.2% (2019).

PDO: Enhance tax revenue mobilization.

Corporate commitment on gender: sample result indicators in actual operations

25

1. Female civil servants recruited through the revised process under the Civil Servants Law, as a fraction of the total, rises from 0% (2017) to 15% (2019).

2. Number of property documents recognizing women as full or part owners of land rises from 500 (2017) to 150,000 (2021).

3. Share of pregnant women receiving extended maternal and postnatal care package rises from 0 (2016) to 15% (2019).

Resources

Title Link

Implementation Completion & Results Report for DPF Operations Click here

Policy Based Guarantees under DPF Click here

Designing a Better DPF Results Framework Click here

26