Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development...
-
Upload
kellen-streets -
Category
Documents
-
view
214 -
download
0
Transcript of Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development...
![Page 1: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/1.jpg)
Quantitative Service Delivery Surveys:Some lessons from schools surveys
Deon Filmer
Development Research Group
The World Bank
Kampala, Uganda
July 29, 2009
![Page 2: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/2.jpg)
Why carry out a Quantitative Service Delivery Survey?
Outcomes are low …
![Page 3: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/3.jpg)
Starting points: Learning outcomes are low
Inequality in TIMSS 2003 Mathematics test scores
200
300
400
500
600
700
Un
ited
Sta
tes
Ko
rea
Hu
ng
ary
Lith
ua
nia
So
uth
Afr
ica
Ch
ile
Ma
lays
ia
Bo
tsw
an
a
Ru
ssia
nF
ed
era
tion
Ira
n
Ph
ilip
pin
es
Jord
an
Eg
ypt
Ind
on
esi
a
Quintile 5 Quintile 1 AverageSource: Analysis of TIMSS 2003 database
![Page 4: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/4.jpg)
Starting points: Learning outcomes are low
Inequality in SACMEQ 2000 Mathematics test scores
200
300
400
500
600
700
Se
ych
elle
s
Ma
urit
us
So
uth
Afr
ica
Bo
tsw
an
a
Na
mib
ia
Sw
azi
lan
d
Le
soth
o
Ug
an
da
Ke
nya
Mo
zam
biq
ue
Za
mb
ia
Ma
law
i
Ta
nza
nia
Quintile 5 Quintile 1 AverageSource: Analysis of SACMEQ 2000 database
![Page 5: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/5.jpg)
Why carry out a Quantitative Service Delivery Survey?
Outcomes are low …
… is it a lack of money?
![Page 6: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/6.jpg)
Public spending is not enough to improve outcomesPattern across countries
* Difference in logs (x100) form rate predicted by GDP per capitaSource: WDR 2004
![Page 7: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/7.jpg)
Public spending is not enough to improve outcomesPattern across primary schools in Mauritania
![Page 8: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/8.jpg)
Similar changes in public spending can be associated with vastly different changes in outcomes…
Source: WDR 2004
![Page 9: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/9.jpg)
…and vastly different changes in spending can be associated with similar changes in outcomes.
![Page 10: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/10.jpg)
How to assess the lack of association between spending and outcomes?
Public spending benefits the rich more than the poor– Expenditure incidence analysis of public spending for
diagnosis Lack of demand by households
– Impact evaluation of programs to promote demand Money fails to reach frontline service providers
– Public expenditure tracking surveys (PETS) Poor quality services
– Quantitative Service Delivery Survey (QSDS) e.g., absenteeism, time on task
![Page 11: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/11.jpg)
Public spending benefits the rich more than the poor
0
5
10
15
20
25
30
35
40
Tanzania1993/94
Coted'Ivoire1995
Uganda1992/93
Guinea1994
Brazil1997
Malawi1994/95
Indonesia1998
Kenya1992/93
SouthAfrica2000
Poorest quintile Quintile 2 Quintile 3 Quintile 4 Richest quintile
Expenditure incidence of public spending
![Page 12: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/12.jpg)
Lack of demand by households0
.2.4
.6.8
1P
roba
bilit
y o
f en
rollm
ent
1 2 3 4 5 6 7 8 9 10Decile
Non-recipient Recipient
Impact of a scholarship program on girls’ enrollment in Cambodia:Enrollment probability among recipient and non-recipient girls by economic status decile
Source: Filmer and Schady (2008)
Impact of a conditional cash transfer on girl’s and boy’s middle school enrollment
Impact of demand-side programs
![Page 13: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/13.jpg)
Disbursed public spending on school grants that actually reach schools
Percent
GNI per capita (2000)
GNI per capita PPP
(2000)
Ghana 1997/98 51 330 1880
Kenya 2004 (secondary school bursary funds)
78 250 810
Madagascar 2002 88 2050 4610
Peru 2001 (utilities) 70 / 97 670 2280
PNG (2001/2002) 72 / 93 280 510
Tanzania 2002-2003 62 270 1250
Uganda 1991-1995/2001 <20 / 80
Zambia 2001 (discretion/rule) 24 / 90 320 740Ye and Canagarajah (2002) for Ghana; Republic of Kenya (2005) for Kenya; Francken (2003) for Madagascar; Instituto Apoyo and World Bank (2002) for Peru; World (Bank 2004)
for PNG; MOF, Government of Tanzania (2005) for Tanzania; Reinikka and Svensson (2005) for Uganda; Das et al. (2002) for Zambia.
Percent of school grants that actually reach schools
![Page 14: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/14.jpg)
Classic approach to analyzing education outcomes…
Inputs School Outputs / Outcomes
Money
$$
Quality of public services
… QSDS are a way to get inside the “black box” of service delivery at the facility level
![Page 15: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/15.jpg)
What are Quantitative Service Delivery Surveys?
Take the facility as the unit of analysis– Could be complemented with a household/users survey
Collect quantitative information about– Physical infrastructure– Staff characteristics– Income and expenditures– Governance and management– Characteristics/Quality of service provision– Outcomes
![Page 16: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/16.jpg)
Two different sets of surveys
Indonesia:– December 1998: Early days of economic crisis …
were schools feeling any impact– April/May 2000: Longer-run school-level impacts
of the crisis, decentralization looming PNG
– April/May 2002: little knowledge about the status of services in PNG; particular interest in decentralization; explicit concern about expenditure tracking
![Page 17: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/17.jpg)
Some lessons from experience, with a focus on two different sets of surveys
Indonesia:– 600 schools– 5 purposively selected provinces– 15 districts (40 schools per district)
PNG– 220 schools– 8 purposively selected provinces– 2 districts (10 schools per district)
![Page 18: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/18.jpg)
Activity structure
Indonesia:– Close collaboration with research department of ministry of
education.– Ministry staff served as full partners in pilot/questionnaire
development; served as regional survey supervisors.– Gave the survey some legs within the ministry, enabled substantially
lower costs … but cost in terms of capacity and experience.– Study conceived of as stand alone survey, with
Ministry/policymakers as primary audience. PNG:
– Partners with National Research Institute, an independent agency– Overseen by “working group” with various government, NGO, and
other representatives.– Little hands on input from Ministry of Education.– Study conceived of as a part of WB Poverty Assessment.
![Page 19: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/19.jpg)
What worked well Indonesia
– Trends in enrollments at the school level Non-conventional wisdom result that enrollment impacts were
mainly urban and at the secondary level; and in non-private/non-secular schools.
But difficulty: enrollment levels/trends … not enrollment rates.– Perceptions of impact of crisis
Identified “general impact” and “school functioning” as two main impacts (exploratory PC analysis)
– Status of crisis-relief government programs (scholarship and grant programs)
Schools grants: Coverage; use; interesting substitution between grants and other sources of government (especially local government) sources of funding
Scholarships: Coverage (among students)– Trends in charging of fees
![Page 20: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/20.jpg)
Indonesia 2000: Sources of school funding by grant receipt and public/private status
0
1000
2000
3000
4000
5000
6000
7000
8000
Public-Received
Grant
Public-NoGrant
Private-Received
Grant
Private-NoGrant
Grant Local Central
Primary schools
0
20000
40000
60000
80000
100000
120000
140000
Public-Received
Grant
Public-NoGrant
Private-Received
Grant
Private-NoGrant
Grant Local Central
Junior Secondary schoolsIn public schools, local government spending adjusted in response to grant
No adjustment in private schools
Substitution between grants and local government funding
![Page 21: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/21.jpg)
Zambia 2001: Effect of a 100 Kwacha increase in expected and unexpected school grants on household expenditures on education
-50
-45
-40
-35
-30
-25
-20
-15
-10
-5
0
Expected Unexpected
Kw
ac
ha
Household spending falls by about 45 for each additional 100 Kwacha spent on anticipated grants
Substitution between grants and household spending
Source: Jishnu Das, Stefan Dercon, James Habyarimana, Pramilla Krishnan (2004)
![Page 22: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/22.jpg)
What worked well PNG
– Descriptive status of schools (very little prior information)
– Good documentation of delays in subsidies / teacher pay
– Reasonable assessment of teacher absenteeism (pre-announced window for visit)
– Good data to construct “ghost teacher” estimate (with substantial effort in matching to government payroll records)
![Page 23: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/23.jpg)
Delay in ability to use subsidy: PNG 2001
0
20
40
60
80
100
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
Accessible Remote
Percent who received any subsidy
0
5
10
15
20
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
Accessible Remote
Weeks delay
Note: Q1,Q3=National, Q2,Q4=Provincial
![Page 24: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/24.jpg)
Absence rates among teachers and health workers
Note: Surveys were all fielded in 2002 or 2003. Sources: Chaudhury et al (2006) except for PNG, World Bank (2004) and Zambia, Das et al (2005).
0
10
20
30
40
50
Bangladesh Ecuador India Indonesia Papua NewGuinea
Peru Zambia Uganda
Primary schools Primary health facilities
![Page 25: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/25.jpg)
PNG 2002: Depletion of the effective supply of teachers
Source: PESD 2002.
85
72 68
100
0
20
40
60
80
100
Teachers onpayroll
Minus "ghost"teachers
Minus absentteachers
Minus schoolsclosed "for
lack ofteachers"
Results from QSDS:Effective supply of teachers
![Page 26: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/26.jpg)
Percent of time officially allocated to schooling; when a teacher is present; and spent in teaching and learning activities
Beyond absenteeism: Effective supply of teaching
Sources: Egypt, Yemen and Lebanon from Lane and Millot (2002); Tunisia, Pernambuco, Morocco and Ghana from Abhadzi, Millot and Prouty (2006); Cambodia from Benveniste, Marshall and Caridad Araujo (2008); and Laos from Benveniste, Marshall and Santibanez (2007).
![Page 27: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/27.jpg)
Investigating accountability in education service deliveryPNG 2002: Teacher absence declines with parent and community involvement
0
5
10
15
20
25
30
0 1 2 3 4 5
Index of parent and community involvement
Pe
rce
nt
ab
se
nt
Source: PNG PESD 2002.
![Page 28: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/28.jpg)
What was harder
Indonesia:– Trends in overall school incomes—never clear we had full
picture (what we did have was worrisome, especially for private schools)
– But, incredibly complicated system … is this worth doing when the system is so complex?
PNG:– Complex funding system … but able to track some specific
payments (school subsidies)– But … school financial data very spotty
only about half of the schools had documentation about spending, half about receipts
Only 30% of schools had both expenditures and receipts documentation
![Page 29: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/29.jpg)
Funding education in PNG2001, million Kina
Leave fare
Payroll Salary(284)
Recurrent(153)
Development(91)
Salary Subsidy Q1&Q3
(40)
in kind
Leave fare(1) Subsidy Q1&Q3
Subsidy Q2&Q4 Subsidy Q2&Q4 (21)
Grants Grants (15)
Donors Grants & donationsNGOs (66)
(68)
Provincial Government and Administration
Dept. of Finance
and Treasury
National Dept. of
Education
Non-teaching staff
Parents
Bank account (school/ BOM/ Joint)
Project fees
School fees
School
Bank account
Teachers
Contractors
Source: Based on information collected during the PESD 2002 survey.
Q1,3
Q2,4
![Page 30: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/30.jpg)
What I would think twice about doing again
Enrollment trends (unless have information on universe of schools and on population trends by area)– Hard (time consuming) to collect, hard to interpret
Too many instruments– PNG had 9 instruments, 7 at the school level.
Non-representative/random sample of parents
![Page 31: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/31.jpg)
Survey instruments in PNG:
– School (head teacher); – teacher roster; – select teachers; – data appendix; – grade 5 teacher; – board of management member; – parent; – District Education Advisor; – Provincial Education advisor.
![Page 32: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/32.jpg)
I would think (very) hard about what financial data to collect
The more specific the better– But even there, school officials often don’t associate specific
transfers to “official” name Anything more than tracking a clearly defined transfer
is very hard. Even that is hard:– missing information at schools; – missing records at provincial level; – defining the “base”
Official declarations in Government Circulars Budget disbursements School level expectations
![Page 33: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/33.jpg)
What I would never do again
Data entry using a package not designed for that purpose
Sophisticated survey/tracking exercise in a country where policy environment not conducive to use information
![Page 34: Quantitative Service Delivery Surveys: Some lessons from schools surveys Deon Filmer Development Research Group The World Bank Kampala, Uganda July 29,](https://reader035.fdocuments.us/reader035/viewer/2022081516/5518cba2550346a61f8b5a52/html5/thumbnails/34.jpg)
Thank you!