The Science of Delivery:

Impact Evaluation Results and

Lessons from RBF

Dinesh Nair, Benjamin Loevinsohn and Ifelayo Ojo

Learning from Implementation

The Why and How of

Top Ten Lessons • Review of ongoing PBF operations suggested

some consistent challenges & useful lessons

• Focusing on the most important lessons will

facilitate learning by other teams

• Selection of the Top Ten was done by TTLs

and RBF experts involved in a portfolio

review of eight ongoing RBF schemes

2

Lesson #1

“Show me the money!”

- Make timely payments

• Over 70% of projects have had issues with

making payments on time

3

Outpatient visits in Nigeria declined when

payments were interrupted

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Dec

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Nov

Dec

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Adamawa

Nasarawa

Ondo

Payments

interrupted

4

Lesson 1: “Show me the Money”

Make Timely Payments

Recommendations

Track time required for payments to reach facilities

Establish explicit standards for maximum allowable

delay (2 or 3 months)

Provide resources up-front to health facilities: (i)

gains their confidence; (ii) indicates PBF has

started; and (iii) provides investment funds

5

Lesson #2

“Keep moving the goalposts!”

Continuous Quality Improvement

(CQI) implies Changing the

Quality Indicators

• Many facilities make rapid improvements in

quality and then plateau

6

0

20

40

60

80

100

Mwaro

Muramvya

Kirundo

Cibitoke

Buja-Rural

Kayanza

Ngozi

Makamba

Rutana

Bubanza

Bururi

Gitega

Karuzi

Muyinga

Ruyigi

Cankuzo

Buja-Mairie

Quality scores by province

Average quality score plateaued after 3

quarters in Burundi

7

Lesson 2: “Keep moving the goalposts!”

Continuous Quality Improvement (CQI)

implies Changing Quality Indicators

Recommendations

Quantified quality checklists should emphasize

process over structural indicators

Revise checklist every 1–2 years to emphasize

continuous improvement

Include: 1) observation of patient-provider

interaction; 2) records reviews (need standard

records); 3) Vignettes; 4) Exit interviews, etc.

Invest in training supervisors

8

Lesson #3

“To be discerning, We keep

on Learning”

• RBF as a tool allows for experimentation,

learning and course-correction

9

What did Zimbabwe do?

Quantitative analysis

Qualitative analysis

Design demand and supply-side RBF (urban pilot)

Targeted urban maternal voucher (demand-side)

Pay-for-quality RBF

Reviewed pricing

Improved quality checklists

10

Operational research studies should be budgeted

Carry out process evaluations and use results for

implementation dialogue and decision-making.

Participate in RBF portfolio reviews – look for new

ideas

Lesson 3: To be discerning - Keep on

Learning

11

Lesson #4

“Money is NOT the root of all evil, Lack of Money is” – Worry about Financial

Sustainability

• PBF should not be made to substitute

deficiencies in national health financing

12

PBF sustainability requires wider health

system reforms

Burundi PBF is

combined with FHC

Government pays almost

half–1.4% of budget

allocated

Common pool for

development partners– two

decreased contribution

Cumulative deficit of

almost US$ 8M

Rwanda PBF health

reforms were part of

larger reforms CBHI reform

HR reforms

13

Where is the PBF money going?

14

35%

37%

39%

40%

45%

11%

10%

10%

9%

5%

16%

16%

12%

13%

10%

29%

31%

31%

33%

37%

8%

7%

7%

5%

3%

Q2 2012

Q3 2012

Q4 2012

Q1 2013

Q2 2013

Intervention HFs Intervention DMO Hospital

Control 1 HFs Control 1 DMO

Zambia: Aggregate consumption of PBF funds from Q2, 2012 – Q2, 2013

Where is the PBF money going?

15

Recommendations

Track overall PBF expenditures and understand where

most PBF funds are going

It is easier to increase tariffs than decrease them. This

suggests starting with relatively lower tariffs and

increasing them

Insist that Government(s) live within an envelope

budget.

Be careful about increasing expenditure untied to

improved performance.

Lesson 4: “Money is NOT the root of all

evil, Lack of Money is” – Worry about

Financial Sustainability

16

Lesson #5

“Math-Phobes of the World Unite!”

– Use your data

• Data is vital but under-utilized, despite a lot

of effort invested into collecting, verifying and putting payment data on the web

17

Internet applications with public front-

end displaying performance & financial

information

18

Burundi

Benin

Nigeria

How Zambia uses its data

• Quarterly analysis

• Extensive analysis by expert

o Diff-in-diff analysis of trends using HMIS to compare

across the 3 study arms

RBF vs. Additional financing

Coef 0.904 0.815 0.696 12.944 2.220 -9.316 -2.783

p-value 0.045 0.231 0.229 0.002 0.019 0.735 0.024

RBF vs. Control

Coef 1.174 1.954 1.586 7.850 2.243 -39.929 -2.761

p-value 0.005 0.002 0.011 0.055 0.031 0.158 0.011

Attendance

outpatient

total (calc)

Immunised

fully <1 year

new

Antenatal

1st visit

before 20

weeks

IPT 3rd

dose to

pregnant

woman

Postnatal

care within

6 days

Attendance

Family

Planning

total (Calc)

Delivery by

skilled

personnel

19

Lesson 5: Math-Phobes of the World

Unite! – Use your data

Recommendations

Create reliable systems for data collection and

compilation

Ensure consistent flow of PBF data with minimal

reporting burden on health workers

Make it somebody’s job to analyze data

Deploy software to help with data analysis

Crowd-source analysis!

Attempt to triangulate all available data sources to

determine trends in utilization and outcomes

20

Lesson #6

“Understand your Customers”

-Demand side issues may be under-appreciated

• There is need to understand root-causes of

the problem RBF is trying to solve.

21

Even with large improvements in utilization

since PBF was introduced, absolute

coverage levels remain low in Nigeria

Barriers to service utilization in two Nigerian LGAs:

Transportation Challenges

Variable & unpredictable fees for Services and Drugs

Social and cultural Barriers

Proposed solution: A voucher scheme to improve service uptake

Continuous cycle of learning and responding to help households overcome service utilization barriers and improve health outcomes.

22

Lesson 6: Understand your customers –

Demand-side issues may be under-

appreciated

Recommendations

Consider demand-side issues if coverage levels remain

low in spite of PBF (even if PBF has made a big

difference).

Understand barriers to access – cultural, social and

financial barriers through household surveys; focus

groups, key informant interviews, etc.,

Options for addressing demand side include: (i) closer

work with community structures; (ii) CCTs; (iii)

vouchers; (iv) BCC through facility staff or NGOs

23

Lesson #7

“Away with Flat-Liners”

- Identify poorly performing regions & facilities. Do something about them!

• Detailed understanding of predictors of

success and failure is required

24

There are clearly facilities that are NOT making

progress i.e. “flat-liners”

Non-determinants

• Number of staff

• Remoteness

• Qualification of in-

charge

• Business planning

-

20

40

60

80

100

120

140 Positive deviants

• Community engagement

• Management capacity of in-charge

Institutional Delivery in Adamawa health facilities,

normalized by 100,000 population

25

Lesson 7: Away with “Flat-Liners”

Recommendations

Use existing data to identify facilities that have not

improved or are not performing well

Carry out quantitative and qualitative studies (e.g.

Nigeria and Zimbabwe)

Consider the kind of technical support needed for

the poorly performing facilities and re-examine

what is currently available.

26

Lesson #8

“You can run but you can’t hide”

– Worry about the politicaI threats to PBF

• Even successful RBF schemes face political

pressures that threaten their sustainability

27

Zambia RBF Pilot

RBF PIU not mainstreamed into MOH structure

Institutional changes in MOH

Key champions moved

28

Watch for winners and losers

Recommendations

Political economy expertise should be engaged to

study factors important for scale-up and

sustainability of RBF, to figure out what can be

done differently.

Institutionalizing roles such as purchasing and

project implementation within government

structures may help improve buy in

29

Lesson 8: Worry about the political

threats to PBF

Lesson #9

“Even the Best Laid Plans can go wrong”

– Analyze your Project Design to Ensure Assumptions

Remain Valid

• Understand key-determinants of the PBF program and the linkages between them to produce results

30

Benin

Bureaucracy makes it

difficult for facilities to

spend PBF incentives

May reduce health worker

motivation to perform

User-fees easier to spend

High fees may decrease

healthcare utilization

31

Challenges with health facility autonomy

in Benin and nested PBF in Afghanistan

made implementation different

Afghanistan

Unique environment of

performance agreement

with NGO

Ongoing security concerns

PBF overlaid on PPA

Recommendations

Periodically review the progression of elements in your

theory of change, not just what you are trying to

achieve, but how you get there

Useful analyses for every project include:

What is additional percentage of PBF to health workers'

take-home pay? Are they motivated by this amount?

Balance of supervision versus autonomy at the health

facility level

Resources available at the front-line for health facilities

32

Lesson 9: “Even the Best Laid Plans can

go wrong” – analyze project design to

ensure assumptions remain valid

Lesson #10

“KISS – Keep Impact Studies Straightforward”

- Avoid Impact Evaluation Questions that are TOO Subtle

• It is very difficult to assess differential impacts when there is minimal variation between experimental groups

33

0

600

1200

1800

2400

1* 2 3 4 1 2

2012 2013

Children immunized

ControlTreatment

The RBF intervention and additional

financing group in Benin are looking the

same

No clear difference is

emerging

Another control group

with no added finances

has been included for

counterfactual

34

0

2000

4000

6000

8000

1* 2 3 4 1 2

2012 2013

Assisted delivery

Control

Treatment

Experimental & Control groups in Nigeria

– try hard to be explicit on differences !!!!

35

Recommendations

Assess in advance if experimental variations are

likely to make a difference in outcomes.

Work with the IE team from the start

Ensure counterparts fully understand research

questions, and are convinced of utility.

36

Lesson 10: “KISS – Keep Impact Studies

Straightforward” Avoid Impact Evaluation

Questions that are TOO Subtle

A ship in the harbor is safe – but that is not

what ships are built for. -John A Shedd (1928)

Implement, learn and share

lessons!