Content

• Added value in Structural Funds• PCM / LFA• Theory of Change• Output based subsidies• Outcome mapping• Innovation Pipelines• Conclusions

1

Performance based financing

• There is a tempting idea to skip all the problems of input based financing (paying for effort - inputs and activities) and focus only on delivering results.

• What is important is the result delivered, not the way how it was created. Let‘s directly link financing to outputs/outcomes delivery.

• This part of the seminar investigates this idea.

Structure of presentation

• Introduction• General Output Based Aid model• Case study: Czech Kindergruppe output

based call• Group work• Conclusions

Linking finance to performance - termsFinancing linked to

Outputs Results

Incentives

primarily for

Governments

Service providers / Project promotors

People / Clients / Target groups

PBI - Performance based incentives, P4P – Pay for perf., RBF – Results based financing,PBC – Performance based contracting

CCT - Conditional cash transfers

OBA – Output based aidOBF – Output based financing

PBC – Perf. based

contracting(health)

Vouchers

CoD – Cash on deliveryPforR – Program for results

DLI – Disbursement-linked indicators

Source: Author, based on Performance-Based Financing Toolkit (World Bank)

OBA – focus of this part of the seminar

Key messages to take from the ToC part

• It requires significant expertise to develop a good Theory of Change

• It is a static approach, that assumes the world remains stable for a while

Output Based Aid: A very brief intro

• Output based aid is the contracting out of service provision to a third party with payments made after the delivery of specified outputs.

• It involves formulating a theory of change, but the difference from previous approach is that the ToC is done by the PMO to prepare a blueprint/template for standardized intervention.

• Delivery partners cannot decide on objectives, outcomes and outputs, as these are given.

Output Based Aid: A very brief intro• OBA approach is suitable for „enhancer” PMOs:

For basic, stable, large volume, well-understood services Often used to close the gap between ability to pay and the actual

cost to ensure consumption of certain services at socially optimal level

ToC is used to make clear how output contributes to longer term outcomes

• In Systems thinking words, OBA approach is suitable for interventions with predictable demand with low variability.

Output Based Aid: A very brief intro• As an alternative, enhancer PMOs can also stick to an

input financed approach: Keep ToC simple with a straightforward planned output-outcome

intervention logic

• In both cases, support “Toyota Production System”- like service delivery mindset.

• Again, this is a static approach.

The GPOBA model

• The Global Partnership on Output-Based Aid (GPOBA) is a multidonor trust fund administered by the World Bank. Its purpose is to fund, demonstrate and document OBA approaches to supporting the sustainable delivery of basic services to those least able to afford them.

• Many experience from development countries, mainly used for basic infrastructure (electricity, water supply), but also in health and education sector.

• Universal principles enable OBA use in the ESIF.

Output Based Aid: Six Core Concepts

1. Targeting of subsidy: it is clear to whom, why, and for what the subsidy is provided. End-users are intended to benefit directly.

2. Accountability: by paying only after delivery of pre-agreed and verified outputs, risk is shifted to the service providers who are accountable to both their clients and the donor (PMO).

3. Innovation and efficiency: Focus only on outputs leaves service “solutions” partly up to competition of providers.

4. Using incentives to serve the needed: OBA encourages service providers to serve those they might otherwise disregard.

5. Output verification and monitoring: monitoring of outputs is easier and more precise in an OBA scheme as the payment is made only for independently verified outputs.

6. Sustainability: OBA focuses on affordability for users (pay user fees they can afford and are willing to pay), cost recovery for service providers and future sources of funding.

Output Based Aid: Typical Scheme

Donor/ Government

Independent Verifying Agent

Target GroupService Provider

Implementing Agency

Selection of Agency, Funds

provision

Contracting SP

Services provided

(Fee for service)

Service delivery

verification

Service delivery

confirmation

Output based payment

Output Based Aid: Typical Scheme

EC / Government

? IVA

Target GroupDelivery partners

PMO

OP

Call for projects

Services provided

(Fee for service)

Service delivery

verification

Monitoring report

Payment claim disbursed

Designing OBA Scheme – 8 steps

Source: GPOBAThis can be visualized in a form of Theory of Change!

Designing OBA Scheme A - Determinign the output

What service is to be provided? Define the output units.

Suitable are services that are:

Keep in mind possible perverse effects of narrow definition of the service (gaming, creaming…)

Clearly definable both in terms of quantity and quality

Easy to verify and measure

As close to the desired result as is feasible

Designing OBA Scheme B – Reaching the targer population

How to determine who should have access to the services? More targeting strategies:

Cost – benefit of targeting, nature of the service and other policies in place should be considered.

Geographic targeting

Self-selection targeting

Income testing targeting

Community based targeting

Combination

Designing OBA Scheme C – Choosing a Subsidy Form

Subsidy should fill the gap between the total cost of a service and the user‘s ability to pay the cost.

One-off subsidies

Transitional subsidies

On-going subsidies

Designing OBA Scheme D – Determining Value of the Subsidy

Subsidy = unit cost – ability of user to pay.

Unfortunately, both actual costs and ability to pay is often unknown at OBA design stage.

Ex-ante approximation

Competitive process

Use more sources, verify independently, benchmarking.

Let the market decide, danger of „the winner‘s course“.

Designing OBA Scheme E – Linking Outputs to Subsidy Disbursements

As the subsidy is disbursed after the delivery of outputs, consider the timing of outputs and payment verification and the ability of service providers to pre-finance outputs. Advance payments could be necessary.

Designing OBA Scheme F – Organizing the Institutional Framework

Several questions to ask:Who are the expected Service Providers?Do SP have technical expertise to deliver service?Do SP have sufficient capacity for our policy goals? Are SP private, public, NGO? Are SP ready to take the risks? Is the service subject to governmental regulation?How the results will be verified? (IVA?)

Some answers may require PMO to do additional actions (capacity building, regulatory initiative).

Designing OBA Scheme G – Evaluating and Mitigating the Project RisksRISK MITIGATION

Performance Risk: Output not delivered in agreed terms.

Verification controls.Capacity building before implementation.

Payment Risk: Output delivered, but payment delayed.

Clear disbursement procedures, PMO internal procedures quality.

Demand Risk: Wrong estimate of demand for the service.

Demand studies, awareness rising campaign.

Unit Cost Change Risk: Factors out of SP control.

Careful unit cost calculation and ex-ante evaluation.Mechanisms allowing adjustments.

Collection Risk: Target group not able/willing to pay their part.

Willigness-to-pay surveys

Regulatory Risk: Change in regulation of the service.

Quality of regulation check in design phase. Good communication between the PMO and the regulator.

Designing OBA Scheme H – Organizing Monitoring and Evaluation

• Monitoring and data availability is integral part of the design.

• Available info on costs and outputs enables CBA.• Reporting burden is on Service Provider, verification on

Independent Verifying Agent.• Linking disbursement to monitoring makes inevitable

management attention. • Client satisfaction surveys could serve as quality control.• Evaluation still necessary to confirm relevance and

causal link between output delivery and results.

OBA Suitable when…

Source: GPOBA

Output Based FinancingCase study, part I:Czech Kindergruppe output based call

Linking Finance to Performance with Output Based Financing

The Czech case study:The Problem to be solved• Problem definition:

Too many parents (mostly women) for too long time leave the labour market because of children.

• Consequences:• Parents loosing their competences on the labour market.• Part of skilled labour force is absent on the labour

market.• Higher demands on social system.• Lower income of the national budget.

• Policy objective:More (women) sooner back to work.

The Problem to be solved

• Policy objective:More (women) sooner back to work.

• General solution:Parents have to have the need to re-enter the labour market and the possibility to satisfy this need. Need is affected by

• Economic parameters: Is it economically worth it to start to work (function of salary, work-related costs, taxes and social benefits)

• Motivation for personal development• Societal pressure: „Only bad mothers start to work early“

Possibility is affected by• Affordable child-care services in acceptable quality• Availability of jobs (esp. part time jobs).

The Problem to be solved

• Policy objective:More (women) sooner back to work.

• General solution:Parents have to have the need to re-enter the labour market and the possibility to satisfy this need. Need is affected by

• Economic parameters: Is it economically worth it to start to work (function of salary, work-related costs, taxes and social benefits)

• Motivation for personal development• Societal pressure: „Only bad mothers start to work early“

Possibility is affected by• Affordable child-care services in acceptable quality• Availability of jobs (esp. part time jobs).

Focus of the „Kindergruppe“ intervention

Intervention design

• Main objective To enhance participation in the labour market

by reducing the barriers for parents with small children by supporting creation of employer-based childcare facilities (non-occasional and non-short-term care) – „Kindergruppen“.

Intervention design

• Identified obstacles: Lack of know how on the side of employers. Lack of legal certainty: such facility is possible to run,

however it is not explicitly defined in the Czech legislation and there are divergent attitudes of relevant authorities (hygiene, fire department, building authority).

Economic parameters: full costs at level of 7K – 10K CZK per child/month, when most parent could afford such service for max. 3 – 4K CZK.

Low trust of some parents due to lack of „institutional guarantees of service quality“.

= 27 CZKMedian wage 17K CZK netto

Intervention design• Actions taken

Preparation of ESF Funded output based grant scheme in order to

• Provide employers a blueprint for such a facility• Provide funding for launch and initial phase of operation• Piloting of output based grant scheme in ESF.• Current situation: 65 Kindergruppen supported, first welcomed

children at the beginning of September 2013

• Adopt Kindergruppe legislation in order to• Provide legal certainty• Contribute to solution of economic part of the problem by tax

deduction arrangement for employers• Current situation: Law prepared, legislative process interrupted

by the fall of the government, however the law is still a priority of the new government and should be in force from September 2014.

Linking Finance to Performance with Output Based Financing

Intervention design

• Funds allocation CZK 250 million (app. EUR 10 million)

• Timing December 2012 (call for proposals) → June 2015 (projects

completed)

• Basic features Support creation of employer-based childcare facilities (non-

occasional and non-short-term care) – „Kindergruppen“ for children from 6 months of age.

Employer or a consortium of several employers can apply, services must be provided for employees only.

Kindergruppe size 7 – 24 places, when for size 13-24 some additional (mostly hygienic requirements) apply.

…to be continued

Now, time for your Group Excercise

Group Excercise - Task

• Form groups of 3 – 5 people• Expected result from each group:

7 min presentation, that will explain your solution of the call, including. You have 30 min to prepare.

• 1. Provide definitions of used output indicators on which payments are to be based

• Clearly definable both in terms of quantity and quality• Easy to verify and measure• Close to the desired result as is feasible (significant influence of the

project promoteur, but includes „demand side“)

• 2. How would you collect and verify the data on output indicators?• 3. Do you see any risks linked to your indicators? (Perverse

incentives, possibilities for cheating, gaming, parking, creaming?) How you could mitigate them?

Output Based FinancingCase study, part II:Our solution

Output definition

• Long iterative process, discussion both internal and external

• Using theory of change as a tool• We started from complicated scheme and

reduced the number of output types later from initial six to just two.

Linking Finance to Performance with Output Based Financing

Activities

Activities Participation

Behavioral change

Kindergarten places created

NR of places in kindergartensMax 1000 places10k CZK per place

10 M CZK

Organizations involved in project to create child-care services for

employees

NR of organizationsMax 40 organizations

100k CZK per org4 M CZK

Places available

NR of full-time child-places-monthsMax 1000 children 24 months1500 CZK per child-place per

month36 M CZK

Capacity used

NR of full-time-equivalent child-months in kindergarten

Max 800 children 24 months1825 CZK per child per month

35 M CZK Parents work intesity increases

NR of families where work intensity increased due to kindergaten

Max 500 families15k CZK per family

7,5 M CZK

Kindergartens in operation 6 months after the end of support

NR of places used in kindergartens in operation 6 m after

Max 800 places5 625 CZK per place

4,5 M CZK

Evaluation of the JAP, identification of success factors, best practice identification and disemination...

Evaluation report written, published and circulated1 report3 M CZK

JAP – Child care servicesBudget 100 M CZK = 4 M EUR

Initial thoughts

Kindergarten places created

NR of places in kindergartensMax 1000 places10k CZK per place

10 M CZK

Organizations involved in project to create child-care services for

employees

NR of organizationsMax 40 organizations

100k CZK per org4 M CZK

Places available

NR of full-time child-places-monthsMax 1000 children 24 months1500 CZK per child-place per

month36 M CZK

Capacity used

NR of full-time-equivalent child-months in kindergarten

Max 800 children 24 months1825 CZK per child per month

35 M CZK Parents work intesity increases

NR of families where work intensity increased due to kindergaten

Max 500 families15k CZK per family

7,5 M CZK

Kindergartens in operation 6 months after the end of support

NR of places used in kindergartens in operation 6 m after

Max 800 places5 625 CZK per place

4,5 M CZK

Evaluation of the JAP, identification of success factors, best practice identification and disemination...

Evaluation report written, published and circulated1 report3 M CZK

JAP – Child care servicesBudget 100 M CZK = 4 M EUROutputs

Outputs Outcomes

Outcomes

?

Reduction…

The final model

Services offered to parents

Sustainability of the facility expected

Black box

Higher intensity of labour market participation on labour market in families of parents

Paid unit:Place created.

No payment.

Paid unit:Place occupied. Support decreases in time from 100% in Stage I to 50% in Stage III => road to sustainability after the support ends.

No payment.This is assumption in the design, which is to be tested by evaluation to answer the question if this intervention is worth to repeat.

Preparation of the facility (max. 4 months)

Max 22 months of support

Parents participate, services are provided

Stage I(6 months)

Stage II(6 months)

Stage III(6 months)

Indicator name Places created in kindergruppe

Definition Number of places for kids in quality according to a standard available in newly open kindergruppe. – „Maximum capacity“

Unit Nr of places

Reward per unit 24 114 CZK

Primary data source Monitoring report

Verification source On spot control – observation

Reason Should cover substantial part of investment costs.

Possible perverse incentive or other problems

• There may be tendency to a certain size of kindergruppen – based on production curve or on different requirements for different sizes.

• Quality of kindergruppe „just to meet the standards“

Severity of problems Low

Indicator name Capacity of kindergruppe used

Definition Number of full time equivalent places in each month really attended by client children at least from 70%/40%.

Unit Places x months

Reward per unit 5851 CZK (100% in Stage I, 75% in SII, 50% in SIII, if attendance rate => 70%)(60% in Stage I, 45% in SII, 30% in SIII if => 40%)(0%, if attendance rate below 40%)

Primary data source Monitoring report – online attendance sheet

Verification source On spot control – observation, interviews

Reason Should be the main incentive.

Possible perverse incentive or other problems

Dead souls in attendance sheets. Part time children reported as full time etc.

Severity of problems Moderate

How to set the price• Unit cost were calculated on basis of:

• Unit: Place created• Review of previous similar real-cost based ESF

projects• Market research of compulsory equipment

• Unit: Place occupied• Review of previous similar real-cost based ESF

projects• Occupancy 70% threshold was selected after

investigation of sample of Kindergardens attendance (children on average miss 28 % of time).

• Prices, including methodology how they were calculated, were discussed with Payment and certifying authority (CZ Ministry of finance) and European Commission.

Issues with this method• Project selection

– Criteria of selection are based on objective data (e.g. Share of weighted headcount and capacity of the facility, Regional need, Financial health of the applicant etc.)

– Applicant can calculate number of points awarded in the selection process even before they submit application.

– No external experts are needed for the selection process

• Checks– Costs actually incurred by the facility operator is not taken into

account = no verification of any of the accounting documents – The checks must verify only:

• Quantity of attained units• Compliance with the requirements for quality of the facility (e.g. the

educational requirements for caregivers, eligibility of the target groups, meeting hygiene standards etc.)

• Compliance with the rules for public procurement and the rules for state aid

Lessons learned

Some observations:• Preparation of the intervention scheme took almost a year• Low administrative burden is welcomed both from the side of

applicants and managing authority employees• Only in project selection proces we eliminated cca. 150 man-days of

work.• The interest was lower than expected – only some 30 % of available

funds allocated into 65 projects• Complaints about too low unit prices

• We hope in good sustainability rate as obviously other sources of financing have to be involved.

• Only half of applicants are companies – the rest typically hospitals, universities, research institutes or municipalities

Lessons learned

Some observations:• Tendency to create kindergruppen of 12 or 24 places

• This is response to hygienic requirements. • We were not able to tackle the problem of divergent attitudes of

relevant authorities • This should diminish once the law is in force.

• Unfortunately, more than 20 projects stepped down due to insufficient occupancy and thus lack of financing or other reasons.

• Currently 42 running projects with 770 places. • Based on preliminary evaluation findings, 1 place generates 0,25

FTE of job.• The most important reason for parents to participate is perceived

higher quality of child care in comparison to available alternatives.

Main obstacles for beneficiariesAbsence of legal framework

• There is no specific law for private non-profit child care facilities

• Unclear rules for hygienic and construction standards

Responsibility for the results

• Fluctuating demand from parents for such kind of service– Real demand differs singnificantly from the one declared before the project started –

influence of external factors

• Employers lack sufficient know-how (due to complicated legislation)– They would prefer to use supplier fot the whole project – not allowed

Undervalued units

• It was not meant to be 100% compensation of the real costs– For the purpose of sustainability, financial support declines with every new stage

– Facilities are supposed to be co-financed by beneficiaries or parents themselves

44

Challenges for MA

45

• Monitoring instruments • How to check the occupancy in a simple and reliable way?

• Estimated time for occupancy check with actual instruments is 3-4 hours for one project

• How to work with monitoring system not designed for RBM projects?

• Compliance with the rules for public procurement• How to check in projects based on results and using simplified costs (and

why)?

Ongoing evaluation

46

ESF implementation Department provide ongoing evaluation with the aim:

• To find out, how the projects affect employment of people with small children.

• To find out a motivation of employers to create kindergartens.

• To find out conditions of success and sustainability of the projects.

• To analyze economy of the projects (compared with similar projects based on real costs)

• To analyze the need for childcare services and potential demand for such a kind of projects for 2014-2020

Lessons learnedWhat to do next time better:• Deeper discussion with employers (possible applicants) about the design

and unit prices• Eliminate big steps in payment function as they create danger of perverse

incentives:

0%10%20%30%40%50%60%70%80%90%100%0%10%20%30%40%50%60%70%80%90%100%

Payment dependence on attendance rate

Attendance rate

Pay

men

t re

ceiv

ed

At the level of 70% and 40% even a minimal difference in attendance rate heavily affects the payment receivedMore smooth payment

functions may lower the risk of data manipulation.

Recapitulation1. targeting of subsidy: self-selection of the

employers/applicants

2. accountability: by paying only after delivery of verified outputs = equipment according to standards, only paid when capacity is used to a certain minimal level, risk (demand risk) is shifted to the service provider;

3. innovation and efficiency: “solutions” partly up to service provider.

4. using incentives to serve the intended target group: service offered for lower than market price.

5. output verification and monitoring: on-line attendance sheet, on spot visits,

Recapitulation6. Sustainability:• We expect 4 sources that combined should keep the

facility running Our support in start-up = no need to recover investment

costs Tax deduction established by the new law Employers could provide this service as a highly welcomed

benefit to the employees Parents contribution still expected, but at affordable level.

RecapitulationA. Determining the output Occupied places in the facility

B. Reaching the targer population Selfselection (+ geo targeting, due to fund regulations)

C. Choosing a Subsidy Form Transitional subsidy

D. Determining Value of the Subsidy Ex-ante approximation based on benchmarks

E. Linking Outputs to Subsidy Disbursements

Preparatory phase and Investment costs recovery „place created unit“.

F. Organizing the Institutional Framework Implementing Agendy = Managing AuthorityNo IVA => Verification by MA

G. Evaluating and Mitigating the Project Risks

Not explicitly mentioned

H. Organizing Monitoring and Evaluation On-line attendance sheetsParalel evaluation project

Recapitulation

• OBA could serve well when the problem has a solution in a form of provision of well known and standardizable services.

• „Enhancer strategy“ oriented PMOs should consider use of OBA.

• OBA will not work when the solution is unknown (innovation needed) or unique at the level of individual projects.

• Preparation of the OBA scheme is demanding, but pays-off in simplified administration during implementation.

Linking finance to performance

Some general issues on linking finance directly to performance

Creaming and parking

Other dimensions of performance?

E.g. what does “a kindergarten place” mean?How to define quality?

Take into account need for different services / costs e.g. for job seekers with disabilities versus young recent graduates

Ideally pay for increments rather than thresholds + combine with milestones (=distancetravelled)

Watch out for abuses!

Prefer long contract durations but take measures to pressure for quality and innovation

Beware!

• As Output based financing can shift the motivation of people towards complience with external regulation only (close to amotivation)

• On the other hand, what is a way to transfer funds to a contractor, need not be translated as a way to manage within the contract Leadership within the contractor can still provide an

autonomy supportive environment and may hence get more results than otherwise would be possible

It may be useful to build in some safeguards as to “how” business will be done

57

Further study materials

Excellent overview of OBA in the health sectorhttps://openknowledge.worldbank.org/handle/10986/17194

Global Partnership for Output Based Aid E-learning coursehttps://www.gpoba.org/node/521

• Sourcebook, pp. 153-174• COP RBM OBA Seminar

? Krakow, Poland, September 2014

Further reading

Now it is good time for questions.

60

• Recap on delivery mechanisms

61

How to realise added value?• Enhancers:

Use output based subsidies for basic, stable services ToC to underpin this or to support an input financed approach for large volume

service provision where it merits the investment Keep it simple otherwise (simple planned output-outcome intervention logic) and

support “Toyota Production System”- like service delivery mindset

• Innovators: Use participatory learning approach, structured as an innovation pipeline ToC can be part of developing and testing a concept (e.g. use multiple theories)

• Solutions managers Use outcome mapping, problem driven iterative adaptation, Vanguard redesign,… ToC to get idea of scope / complexity and for evaluation of a particular case (of

what happened, without concluding this will happen again) NOT as a general plan No use for LFA as an approach here either

• static and conflict avoiding approach with tightly specified work programmes that are reported on with “variance” analysis (hiting the targets) not suited for complexity

• but of course stakeholder analysis and problem analysis remain useful as some form of situation analysis is always required

Intervention scopeNarrow Wide

Theoryofchange

Direct/ linear

Complex

Niche resultse.g. basic services(like ambulances,shelters for the poor)

“Transformative” results

Integrated resultse.g. integrated service delivery (like hospital care, education,…)

Ecosystem resultsService delivery combined with more transformative actions enable wider and longer term socio-economic-environmental results.

e.g. changes in societal/organisational norms, policies, practices (tackling local, unique root causes of issues/problems) via advocacy, networks

© Benedict Wauters / Latitude C&T*Adapted from Ebrahim 2010

Intervention scopeNarrow Wide

Theoryofchange

Direct/ linear

Complex

Niche resultse.g. basic services(like ambulances,shelters for the poor)

“Transformative” results

Integrated resultse.g. integrated service delivery (like hospital care, education,…)

Ecosystem resultsService delivery combined with more transformative actions enable wider and longer term socio-economic-environmental results.

e.g. changes in societal/organisational norms, policies, practices (tackling local, unique root causes of issues/problems) via advocacy, networks

© Benedict Wauters / Latitude C&T*Adapted from Ebrahim 2010

Solutions managers drawing on existing services, asking for and introducing

innovative ones

Enhancersmass producing services

Innovation=

complex process of coming up with

service innovations that some day will be

mass produced