PKH Evidence & Policy Implications: summary of Results...

Investing in Indonesia’s Institutions

for Inclusive and Sustainable Development

PKH Evidence & Policy

Implications: summary of

Results from Impact

Evaluation, Operations

Analysis, and Spot Checks

Diskusi Pokja Kebijakan Monev TNP2K

29 March 2011

1Kebijakan Monev 1

Intro Methods Poverty Health Educ Labor Consumption Policy

Summary of Results from PKH Impact Evaluation

− Methodology

− Poverty

− Health

− Education

− Child Labor

− Consumption

Further evidence from Spot Checks and SMERU qualitative study

Policy going forward: discussion

− Education

− Supply readiness

− Facilitators

− Affiliated providers

Today’s topics

Intro

2Kebijakan Monev 2


PKH was randomized to kecamatan, non-random to households.

Methods

Randomization is effective at ensuring that treatment decisions and response to treatment are orthogonal and uncorrelated. “

”

to which should be added “including unobservable characteristics and unmeasured trends”

With equivalent distributions of characteristics we certain that treated and untreated groups are valid counterfactuals for each other:

“Equivalent distributions of characteristics” are tested with baseline (pre-program) data on observable characteristics.

3Kebijakan Monev 3


PKH was randomized to kecamatan, non-random to households.

The PKH kecamatan and household-selection processes:

Methods

4Kebijakan Monev 4


PKH was randomized to kecamatan, but not 100% according to plan

Beware #1: the PKH kecamatan selection process was contaminated:

− 39 out of 180 (22%) surveyed lottery control kecamatan were found with PKH

− 2 out of 180 (1%) surveyed lottery treatment kecamatan were found to never have received PKH

Methods

5Kebijakan Monev 5


PKH was non-random to households in an observable way

Beware #2: the household selection processes was purposive:

Methods

6Kebijakan Monev 6


Beware #2: the household selection processes was purposive:

Result #1: PKH households worse off compared to even other eligible households

− Younger, larger, female-headed, in agriculture

− Less education, fewer assets, more likely to be BLT and Jamkesmas beneficiaries (but scholarships don’t matter)

− Less connected to village authorities and less involved in governmental groups

PKH was non-random to households in an observable way

Methods

7Kebijakan Monev 7


To continue using full matched-at-baseline sample while incorporating lottery/actual mismatches:

− Use lottery assignment as instrument for actual status

− Control (probabalistically) for those who received and missed incorrectly

To compare treated households to like households in control kecamatan

− Use indices reflecting likelihood of receiving PKH; build indices from observed characteristics

− Control (probabalisitically) for those who would have received had PKH been in their kec

To provide an alternative specification (robustness)

− Difference-in-differences method with two alternatives (xsection or matched panel)

− Provide estimates of behaviors over randomly-receiving or randomly not-receiving hh only

Fixes for kecamatan contamination and non-random household selections

What difference in

differences (DID) does:

subtract from pre- to

post-PKH changes the

same over-time change

in non-PKH areas.

Methods

8Kebijakan Monev 8


Average baseline (2007) expenditure (household average, per capita per month): IDR 199,000− average for all eligible households; eventual PKH households slightly poorer at baseline

Average PKH household increase in expenditure (2007 to 2009): IDR 19,000 (9.5% of baseline)

− this is over and above what similar non-PKH families in control kecamatan experienced.

PKH was enough to move ~700,000 eligible beneficiaries out of poverty (on average)

− March 2009 poverty line: IDR 200,262

PKH share in overall measured poverty reduction could have been 82% (on average) :− 2007→2009: headcount poverty rate 16.6% → 14.1% implies 858,000 fewer poor hh

Average baseline expenditure (2007) for eventual PKH households : IDR 184,000− Average PKH impact on these households (IDR 19,000) = 10% of baseline

PKH moved ~700,000 actual beneficiaries to the March 2009 poverty line− implicit reduction in poverty gap/severity: 1.9/0.74 index points

PKH share in decrease in poverty gap/severity could have been 73%/65% (on average):

− 2007→2009: actual poverty gap 5.1 → 2.5 and actual severity 1.8 → 0.675

PKH generated significant reductions in poverty, poverty gap measures

Poverty

1 Implied reductions in poverty measures

Note: all results approximately the same under the assumption that the full value of an average 2008 PKH transfer (IDR 1.2 million) is used (by a PKH household with an average of 5.2 members) on expenditure items used to calculate total household expenditure. Such an assumption implies an average increase in expenditure of IDR 19,231 per-capita per month.

9Kebijakan Monev 9


Many Healthy Behaviours saw significant and positive increases

Health

2a Significant Health Behaviour Changes

10Kebijakan Monev 10

Intro Methods Poverty Health Educ Labor Consumption PolicyHealth





2a Significant Health Behaviour Changes – Spillovers to non-PKH hhs




2a Significant Health Behaviour Changes – further evidence

Healthy Behaviours benefitted from significant promotion

SMERU Qual Study - Java Barat and NTT; 24 desa/kelurahan); Java Barat accessible; NTT remote

For maternal & child care (MCH), PKH led to ↑

− Posyandu attendance (primarily NTT)

− attendance at Posyandu alternatives (primarily urban)

− awareness of and activity on MCH issues both among beneficiaries and village officials, midwives, posyandu cadres, community leaders, religious leaders, NGOs

− Facilitators crucial: positive encouragement and negative threats

PKH did not coincide with or lead to

− ↑ access to MCH for remote villagers

Factors mentioned: no incentives and/or logistical support for posyandu cadres; absent midwives; poor roads and rainy-season shutdown; lack of public transport

− ↑ community involvement in MCH budget or provision decisions



Not all Healthy Behaviours saw significant increases for PKH hhs

Health

2b Not Significant Health Behaviour Changes



2b Not Significant Health Behaviour Changes – further evidence

Provider quality and remoteness contributed to remaining gaps in health

CHRUI Spot Checks - Java (East, West, Jakarta), South Kalimantan, NTB; 36 kecamatan; 1800 mothers, 108 facilitators, 108 schools, 36 Puskemas,


− ↑ access to puskemas services for remote villagers

Factors mentioned: distance, bad transport, no transport available

− ↑ access to comprehensive service at posyandu (materials and tools; information and copies of records; complete schedule of treatments/interventions)

− ↓fees at puskesmas

− ↑ grievance redress when services rejected or delayed by health service providers.

− full availability of supplies at health service providers

Human capital: dentists, midwives, public health officer, paramedics, nutrionist, pharmacist not always available at puskemsas

Vaccines& Vaccine storage: occasionally missing at puskemsas



2b Not Significant Health Behaviour Changes – evidence in urban or densely populated Java

Provider quality and remoteness contributed to remaining gaps in health



Medium- to long-term health outcomes are mostly showing no changes yet

Health

2c Mostly Not Significant Health Outcomes



But health-seeking behavior has already changed…

Health

2c Mostly Not Significant Health Outcomes – evidence from urban, dense, and agricultural areas



High baseline levels in Education make further increases difficult

Educ

3 Mostly Not Significant Education Outcomes



Low levels of Wage work make further reductions difficult

4 Mostly Not Significant Child Labor Outcomes

Labor



3 Insignificant Education Changes – further evidence

Education supply remained inadequate; advocacy was insufficient

For primary education

− Enrollment did not increase because of previously high levels of attendance

− Facilitators crucial for ↑ in Hours/Attendance: positive encouragement and negative threats

note: SMERU finds more effective facilitators in NTT b/c of reduced range, not reduced caseload

For secondary education

− Enrollment increased only in NTT (again, facilitators crucial)

− Facilitators crucial for ↑ in Hours/Attendance


− ↑ access to 2ndary schools

Factors mentioned: remoteness/distance to schools; lack of public trans; overall cost; early marraige/pregnancy; job demands (harvest)

− ↑ supply of quality at 2ndary schools

Factors mentioned: overcapacity (from BOS); decline in parents’ participation (again from BOS); low levels of teacher education; lack of facilities (libraries, labs, teaching aids, electricity, clean water)

SMERU Qualitative

In education, PKH led to ↑

− Awareness among beneficiaries and village officials, midwives, posyandu cadres, community leaders, religious leaders, NGOs

− Awareness of links from education to employers’ requirements and better future

Educ



3 Insignificant Education Changes – further evidence

Education supply remained inadequate; advocacy was insufficient

CHRUI Spot Checks


− ↑ participation or attendance

Factors mentioned: attendance rates already approx. 100%.; no outreach to those who leave school early; PKH did not cover opportunity costs of secondary school

− ↑ access to 2ndary schools

Factors mentioned: remoteness/distance to schools – (2km/22min on average, 20km/120min maximum to reach with foot or pedal power as the most common mode of transport); no tuition waivers;

− ↑ supply of quality at 2ndary schools

Factors mentioned: low levels of teacher education; overcrowding; lack of facilities (libraries, labs, teaching aids, electricity, clean water)

Educ



Region & Sector trends are encouraging; household trends less so

3,4 mostly not significant Education & Child Labor – further evidence

LaborEduc



Consumption, especially of health goods and services, increases

5 Significant Consumption Increases

Consumption



Consumption, especially of health goods and services, increases

5 Consumption demonstrates significant increases in health seeking behaviour

note: does absence of change in

education spending indicate

anything re: hh constraints to

education?

Consumption


Intro Methods Poverty Health Educ Labor Consumption PolicyPolicy

Education – are PKH benefits enough to expect improvements given costs?

PKH benefits from attention; remaining reforms become clear

SMP/SMA costs per year for poor hh: ~IDR 2.5 million or ~30% of expenditure; PKH households generally did not receive scholarships

PKH family size:5.2 members, 3-4 children needing care



Education – or is it payment timing and program design?


2030

4050

6070

8090

100

Per

cent

(%

)

0 1 2 3 4 5 6 7 8 9 10

Years of Education

quintile 1 quintile 2 quintile 3 quintile 4 quintile 5

Education Attained by IndonesianAge 26-28, Year 2009

Graduated SD/MI

GraduatedSMP/MTs

In the beginning, the 4-per year payment design was changed to 3-per year. As a consequence, there was noPKH payment synchronized with due dates for school fees. These modifications were partially due to the inability of UP-PKH and PT Pos to manage keep the

validation and verification process timely (throughput of forms and data)

Policy



Education – or is it in implementation?


Socialization to affiliated providers was weak

− Providers may not have understood that conditionalities (and compliance monitoring) wasn’t important and the PKH was not just “business as usual” for them

See process engineering assessment and CHRUI for factors behind weak socialization activities

Facilitators were not as effective at encouraging education

− Facilitators and the community generally did not make effort to re-insert drop-outs into the system; facilitators were sometimes too diffuse to be able to visit remote households as soon as drop-outs were noted

See CHRUI for factors behind inability of facilitators to improve education behaviors where they were too disperse

Affiliated agencies did not isolate PKH households for monitoring or assistance

− PKH households generally did not receive scholarships or subsidized transportation

Policy

In reality, all of these issues are at work and all need serious thought

Weakness in one area (form processing) has knock-on effects in other areas (altered payment schedule) which can affect outcomes (inability to recapture drop-outs or improve transition rates)



Relative to other Indonesia SSN, PKH expenditures appear appropriate

1 PKH expenditures show appropriate shares for benefits relative to support operations

WB Social Assistance Public Expenditure Review

Policy



1 PKH expenditures show appropriate shares for benefits relative to support operations

WB Social Assistance Public Expenditure Review

PKH’s operation costs have come down, similar to international examples

Other facilitated programs spending similar amounts on operations

Other programs with similar coverage or similar benefits (Jamkesmas, Scholarships) spend very little on non-benefit operations

− These programs are not facilitated

− These programs do not reach only poor households

− These programs have larger impacts for non-poor households.

− These programs rely on local-level administration (at the facility) to run the program – they are essentially transfers from central government agencies to service providers.

Relative to other Indonesia SSN, PKH expenditures appear appropriate

Policy



Qualitative Studies raised several additional operational issues

2 Validation and updates to household rosters

CHRUI Spot Checks

− 18% households could not be re-verified; 69% had Jamkesmas; only 18% had Scholarships; mismatches between UP-PKH pusat databases and actual recipients;

− 84% households reported demographic changes; updates were not monthly everywhere

Factors mentioned: lack of personnel; lack of forms; power outages; non-functioning MIS (including automatic checks by province or central level); inaccurate BPS data

− Solutions

uji coba (pilot phase): Targeting improvements and recertification strategies; MIS fully functional (and staffed?) everywhere; Form printing and delivery supply chain improvements

Building in data sweeps for mismatches to occur some weeks before payment process begins

Policy



3 Compliance verification and conditioning

CHRUI Spot Checks

− 71%, 68%, 29%, 24%, 19% with BCG, measles, polio, DPT, and hepatitis vaccines (respectively); 2.2% not going to school; and no recorded changes in benefits.

Factors mentioned: mismatch between names of mothers/children reported at local level and in UP-PKH pusat database; forms content, printing, and delivery problematic; lack of incentives (mainstreaming); lack of socialization/enforcement on conditionalities (including to affiliated operators like PT Pos)

− Solutions

sudah: MIS fully functional nearly everywhere; CVS system operational for 85% of hh (@ Jan 2010, increasing lately)

uji coba (pilot phase): Form printing and delivery supply chain improvements (joint with PT Pos and facilitators)

Resocialization and re-examining socialization schedule; audit of socialization process.

sharing lessons with affiliated providers (MoNE, MoRA, MoH); incentive schemes or enforceable MOUs with affiliated providers?


Policy



4 Payment timeliness, scheduling, and payment processing schedules

CHRUI Spot Checks

− 75% report correct payment amounts and 95% received exactly the receipt amount; some districts scheduled 2 or 3 tranches per year instead of 4.

Factors mentioned: CVS and data updates not operating; data mismatch.

− Solutions

sudah : “4 per year” schedule enforced; timing tranches to school fees schedule; MIS fully functional nearly everywhere

uji coba (pilot phase): Form printing and delivery supply chain improvements; CVS system operational for 85% of hh (@ Jan 2010)


Policy



Process Engineering examines in detail the “PKH machine”

5 Socialization & Complaints and Grievances systems

Ayala Consulting (& confirmed in CHRUI Spot checks)

− Socialization delegated to KemenKomInfo not adequately carried out (to beneficiaries and affiliated providers) either in content, frequency, or intensity; no printed materials; officers not able to adequately answer questions; lack of monitoring of socialization process resulted in lingering misunderstanding; PKH beneficiaries not thought elibible for Raskin or Jamkesmas; most info via word of mouth or media reports of malfeasance

− Majority of beneficiaries did not know how to file complaint and were unaware of the CGS; written complaints too costly (fear of losing benefits or faux pas); lack of forms; no response on the small amount of complaints received; MIS system does not automatically respond to incoming grievances or complaints

− Potential Solutions

uji coba (pilot phase): recertification and “waiting lists” in the targeting exercise

Policy




5 Socialization & Complaints and Grievances systems

− Potential Solutions (continued)

DepSos, Bappenas, or TNP2K recapture socialization mandate in order to deliver better socialization, policy, materials & effort

Case Management system: facilitators as “one-stop shop” for all social assistance initiatives for eligible households (April 17-29 – re-visit by int’l consultant)

Affiliated agency MOUs: positive and negative incentives to generate interest and effectiveness in affiliated service providers (PT Pos, KemenKomInfo, MoNE, MoRA, MoH) (April 17-29 – re-visit by int’l consultant)

Review of current CGS procedures in manual and investigation of non-deployment (April 17-29 – re-visit by int’l consultant)

Policy



6 UP-PKH organization and cross-agency organization is ineffective

Ayala Consulting & int’l experts (& confirmed in CHRUI Spot checks)

− Affiliated agencies and local gov’ts agencies consider PKH a burden and do not instruct regional offices to give continuous support; coordination meetings not attended; DepSos lacks authority and/or appropriate agreements to monitor progress (and remedy lack of progress) in affiliated agencies (in 2010, no PKH-related activities were undertaken by KemenKomInfo until earliest August); MOUs inappropriate and unenforceable

− regional coordinators stuck in Jakarta; DepSos lacks capacity/authority to implement a full-scale continuous reform cycle (improvements at the margin); no performance evaluations carried out by UP-PKH; no TA or monitoring available at/from UP-PKH propinsi level (this is inadequate for a country as large and diverse as Indonesia); unclear responsibilities for regional and provincial coordinators; PKH has no specialized personnel for policy/planning, training, inter-agency coordination, M&E and continuous reform.

− Potential Solutions

sudah : Korwil moved back to province


Policy



6 UP-PKH organization and cross-agency organization is ineffective

− Potential Solutions (continued)

Affiliated agency MOUs: positive and negative incentives to generate interest and effectiveness in affiliated service providers (PT Pos, KemenKomInfo, MoNE, MoRA, MoH) (April 17-29 – re-visit by int’l consultant)

Review of current UP-PKH organization and feasible blueprint improvements (April 17-29 – re-visit by int’l consultant)

TNP2K to raise PKH’s profile; point out interest other agencies may have; point out the wealth of data indicating weaknesses in cross-agency collaboration


Policy



Issue #6: Organization within UP-PKH

Placeholder for Ayala’s current


Policy

Investing in Indonesia’s Institutions

for Inclusive and Sustainable Development

PKH Evidence & Policy Implications: summary of Results from Impact

Evaluation, Operations Analysis, and Spot Checks

Diskusi Pokja Kebijakan Monev TNP2K

29 March 2011

PKH Evidence & Policy Implications: summary of Results...

Documents

Transcript of PKH Evidence & Policy Implications: summary of Results...