Measuring the Quality of Education and Health Services:
The Use of Perception Data from Indonesia
Basab DasguptaAmbar Narayan
Emmanuel Skoufias
PRMPR- World BankApril 21, 2009
Motivation & Scope-1
Increasing trend of decentralization of service delivery to local governments Decentralization increases accountability, increases citizen participation and political
engagement, improves public service delivery, allocative efficiency and fiscal administration
Good measures for local government performance necessary for evaluating the impact of decentralization Many of these tools include subjective instruments that gauge citizen perceptions (citizen
report cards, community scorecards, facility exit polls, and citizen satisfaction surveys) , (Amin, Das, and Goldstein, 2007).
Can satisfaction-related questions be valuable in measuring the quality of public services, specifically in health and education?
Motivation & Scope-2
satisfaction surveys are appealing A quick and easy way for policymakers to measure the impact of
governance reforms on government performance, particularly for sectors where measurement of service quality is not easy, provided citizen satisfaction is closely correlated with the actual quality of services.
Less time and labor intensive than facility surveys and public expenditure tracking surveys)
BUT…. there is little consensus on whether citizens’ satisfaction reflects the actual quality of services satisfaction surveys Whether useful and if so How to use it
Motivation & Scope-3
Need better understanding of what factors influence satisfaction What household and community level factors (besides
quality) play a role? Decentralization improved outcomes is based on premises (i) increased accountability of service delivery increases performance and (ii) citizens are able to discern between good and bad gov’t and then influence local authorities.
Concerns w/ Satisfaction Surveys
e.g. Indonesia-GDS2 survey
May get High Reported Satisfaction due to cultural norms or social pressure rather than the superior quality of services.
Absence of a common baseline makes itDifficult to interpret the dataTricky comparing data-points across
regions and countries
Figure 1: High satisfaction with services in Indonesia
Figure 2: Positive perceptions of change in quality of services in last 2 years- Indonesia
Health services Health services
58.1
1.2 0.8
32.2
7.7
0
20
40
60
80
100Satisfied
Quite
Satisfied
Quite
unsatisfied
Not
Satisfied NA
% o
f h
ou
seh
old
s
20.6
3.0 5.8
71.7
0
20
40
60
80
100
Better Same Worse NA
Education services Education services
50.2
30.1
10.71.7
7.2
0
20
40
60
80
100
Satisfied
Quite
Satisfied
Quite
unsatisfied
Unsatisfied
NA
72.9
14.15.8 7.2
0
20
40
60
80
100
Better Same Worse NA
Source: GDS-2 (2006)
When most users claims to be “satisfied” with services...
Average satisfaction with public health and education facilities surprisingly high in GDS-2 Among five options (scale of 1 to 5), option 1 or 2 (somewhat satisfied
or better) chosen by around 90 and 80% of hholds for health and education respectively
Inconsistent with the poor reputation of health and education services in Indonesia, also supported by more objective measures of quality from surveys like GDS-2
Problem not unique to Indonesia, but not universal in surveys either High variation in satisfaction with education and health services in a
number of countries where CWIQ-type surveys have included questions on satisfaction
2006-07 survey in Pakistan showed 35% satisfaction rate for public health facilities; 2007 survey in Sierra Leone showed satisfaction rate of ~40% for public schools
High reported satisfaction in Indonesia probably attributable mainly to cultural norms
Key question: is there useful information content in the Indonesia satisfaction data?
7
GDS-2 survey--1 Survey in May-Sept 06 to assess governance and
local public service delivery in Indonesia collecting data on quality and satisfaction from households,
communities, and facilities
Household sample (8544 hholds) nationally representative – 1068 PSUs (hamlets or dusun), in 89 districts (kabupaten/kota), 267 sub-districts (kecamatan) and 534 villages
Facility data collected from a sample of health and education facilities – guided by which facilities were reported as most frequently used by households Health facility data from the 6 community health centers
(puskesmas) most frequently mentioned by households within each sub-district (kecamatan)
School facility data collected from the most frequently used public elementary school in a village and public junior high school in a sub-district 8
GDS-2 survey--2Thus data on both household satisfaction
and objective measures of facility quality are available for 52% of households using public health facilities and 57% of those using public education facilities.
Sources of possible selection bias Households choosing public facilities may systematically differ
from those choosing other types of facilities Restricting the facility sample to the most frequently used
facilities instead of a random sample of all available facilities can potentially add to the bias
9
Model--1 Simple expectancy disconfirmation model
S= f (Quality – Expectations) Where expectations=g(hh char & experience)
A modified expectations disconfirmation model Expectations play a role in the choice of the type of facility; expectations
proxied by household and community level factors Conditional on the choice of a service provider, reported satisfaction with the
service facility is a function of the actual quality of the service and governance (service is an “experience good” experience it only after choosing it).
Caveats: quality of service measured in terms of inputs in the production of services and NOT outcomes (e.g., achievement scores, z-scores).
Model--2Two-stage Heckman model to correct for selection
bias
1st stage selection equation predicts the propensity of households to use a facility for which objective data on quality are available
2nd stage equation examines how satisfaction (S) correlates with indicators of quality and governance, conditional on the choice of a service provider
Models estimated separately for samples from poor and rich districts, and in a pooled sample Accounts for difference in expectations between the
residents of rich and poor areas Rich districts = GRDP pc > median GRDP pc in sample of
88 districts
11
Defining the satisfaction variable in GDS-2 Useful information can be extracted by focusing on
variations in satisfaction level To exploit the variation, we define our dep. variable as a
binary S, which is =1 for all those who chose option 1, and =0 otherwise
For health services, S=1 for 58% of hholds; for education services, S=1 for 50% of households
Analyzing the determinants of S using two models
Simple heuristic model: correlating household characteristics with satisfaction, separately for health and education Illustrates the range of factors – that service providers have no
control over – influencing satisfaction with services among households
2-stage Heckman selection model: corrects for selection bias in facility choice
12
HEALTH
Health facility choice—1st stageExplanatory vars
S=1 if the household uses a public health facility (puskesmas) for which facility data is available, =0 otherwise
Explanatory variables Expectations proxied by household and community characteristics Other factors likely to affect choice of facility: location of facility
(in village or not), availability of information on corruption and the sources of that information
Wealth, rural-urban designation, and provincial location of district
14
Health facility choice—1st stageResults
Demographic/socio-economic characteristics play a statistically significant role in influencing whether or not a puskesmas is the facility of choice for a household
These characteristics are far more important for hh in poor districts than rich districts
Geographic location: relative to households in Java, households belonging to other regions are more likely to use puskesmas, with some differences between rich and poor districts
Household’s relative position in society (leadership position) or access to information do not seem to matter for choice of health facility
Caveat: these factors explain the choice of a certain type of public health facility for which facility data is available
15
Satisfaction with health-- 2nd stageExplanatory vars
Objective indicators of quality and governance environment Quality : (a) coverage area of puskesmas; (b) types of medical
support available from ancillary facilities; (c) quality of services (human resources and medical supplies); (d) infrastructure
For each category, multiple indicators combined into a single index created using Principal Component
Institutional /governance environment : (a) willingness to complain (voice), (b) govt responsiveness to complaints (accountability), and (c) an index of participation in the administration of health services
16
Satisfaction with health-- 2nd stageResults
Strong correlation between satisfaction and certain dimensions of service quality Namely, support available to the main puskesmas from ancillary facilities, the
quality of service care in terms of human and medical resources (for poor districts); quality of infrastructure has insignificant impact
Thus citizen satisfaction with health facilities, particularly in poor districts, correlated more with the availability of ancillary facilities, human personnel and medicines, rather than facility infrastructure
Institutional and governance indicators correlate with satisfaction, but with rich-poor differences Hholds in poor districts more likely to be satisfied when they participate more
in the administration of health services Hholds that complained about health services less likely to be satisfied with
health services; hholds in rich districts more satisfied when health services are responsive to complaints
All the governance indicators may be endogenous with perceptions of quality and satisfaction
18
EDUCATION
Education facility choice—1st stageExplanatory vars
Sample restricted to households that have at least one child of school age
Dep variable =1 if the household sends a child to a public school for which facility data is available, 0 otherwise; explanatory variables similar to 1st stage model for health
20
Education facility choice—1st stageResults
As expected, a household’s “selection” of a school is the result of a very different decision process from its choice of medical care Demographic and socio-economic characteristics matter less
for the choice of schools than puskesmas Access to information and social status of households seem to
matter for schooling choice and not for the choice of health facilities
Regional location of the household again plays a role: relative to households in Java, households belonging to other regions are more likely to send their children to public schools
Again, the model explains the choice of a certain type of school (public) for which facility data is available
21
Satisfaction with education--2nd stage
Explanatory vars Indicators of school quality : (a) quality of
infrastructure in school; (b) the quality of teaching staff; ; (c) student performance; and (d) coverage of students (enrollments, attendance) by the school
Institutional and governance environment: (a) information available to households about
bribery and corruption in education, (b) willingness to complain against service
providers, (c) provider’s responsiveness to complaints, (d) extent of participatory decision-making in school (e) coverage and implementation of BOS 22
Results from 2nd stage of selection model for satisfaction with public schools
All districts Poor districts Rich districts
Dependent variable: S =1,0 Coef. Coef. Coef.
Participatory mode of management 0.055* -0.043 0.124**
Index of Facilities 0.011 0.029** 0.002
Index of teacher qualities 0.007 -0.010 0.017
Index of BOS coverage 0.017* 0.002 0.027*
Index for student performance -0.009 -0.009 -0.016
Index for student coverage 0.002 0.026 0.001
Responsiveness of the service provider 0.013 0.075* -0.027
Complaints made (Voice) -0.087** -0.145** -0.063
Corruption information -0.100* -0.018 -0.169**
Bribery information -0.149* -0.062 -0.139
Dummy for urban -0.034 -0.015 -0.014
Region dummies (Java as reference):
Kalimantan -0.107** -0.214** -0.002
NTT -0.044 -0.080 -0.095
Sulawesi -0.123** -0.208** -0.051
Sumatra -0.149** -0.432** -0.036
_cons 0.610** 0.676** 0.496**
Satisfaction with education--2nd stage Results
Strong correlation between satisfaction and certain
dimensions of school quality in the pooled sample, but
important differences between rich and poor districts.
Satisfaction levels have significant correlation with: Better infrastructure facilities in schools (e.g. condition of classrooms,
availability of books, library, computers) – for poorer districts only
Participatory decision-making for school’s mission and vision (jointly by
school principal, teachers and the community) – for rich districts only
The coverage of a school by the BOS program and the progress in
implementation of BOS – for rich districts only
24
Satisfaction with education--2nd stage Results
Thus satisfaction in poor districts more influenced by the basic features of a school (e.g. condition of building facilities), whereas in richer districts influenced more by “second-generation” factors (e.g. participatory mode of mgmt) ?
Knowledge of bribery and corruption in education and complaints against schools have significantly negative effects on satisfaction, but again with differences between rich and poor districts
25
Comparing results: health vs. education--1
Key differences in what factors matter for satisfaction
with health and education services
While quality of infrastructure has no influence on
satisfaction with health facilities, quality of school
infrastructure has significant influence on satisfaction with
schools in poor districts
Indicators of quality (availability of personnel and medicinal
inputs) are key determinants of satisfaction in health
services; but teacher quality does not seem to correlate at
all with satisfaction in schools 26
Comparing results:health vs. education--2
For both education and health facilities, a more
participatory management of the facility induces
higher satisfaction. However, differences in how the
results vary across rich and poor areas For health facilities, households in poor districts are more likely to be
satisfied with higher participation in the administration of health services
For schools, households in rich districts are more likely to be satisfied when
management of schools is more participatory or the implementation of a
school-based management system is more advanced
Responsiveness to complaints about facilities improves satisfaction with
health facilities in rich districts only and satisfaction with schools in poor
districts only27
Comparing results:health vs. education--3
Important difference in how the regional
location of a household influences
satisfaction Matters only marginally for satisfaction with health facilities (and
has no effect for poor districts)
But satisfaction with schools likely to be much lower in the
poorer areas of the Kalimantan, Sulawesi and Sumatra regions
compared to the poor areas of Java region
28
Implications of our analysis
29
“Satisfaction” data has important information content
But requires finding meaningful variation in responses and models to account for selection bias and role of expectations in facility choice For health and education, satisfaction with facilities significantly
correlated (in the right direction) with measures of quality, governance and institutional environment of the facility
In many cases, collecting satisfaction data matched with facility level data is not practical Common practices: user surveys/citizen report cards, hhold
surveys
What does our analysis suggest for such “2nd-best” scenarios The design and use of instruments that just measure satisfaction
from hhold surveys, as a proxy for quality of services?
Implications for surveys collecting satisfaction data--1
30
Even if a linked facility survey is not possible, clear benefits in having a satisfaction survey collect as much information on the characteristics of households and communities as possible, including proxies for governance/institutional environment
Random sample administered at household level is likely to yield more representative results in most cases than a survey of a sample of users of a particular type of facility Better for correcting the selection bias arising from facility choice, in
contrast to a survey limited to the users of a particular type of facility Incorporating questions on satisfaction with basic services in household
surveys is becoming more common (E.g. CWIQ surveys combining questions on satisfaction with basic services with hhold and community characteristics)
In some cases where user surveys are the only practical option (e.g. when a service is used by a tiny % of the population), collecting hhold/community level information from users is recommended
Implications for surveys collecting satisfaction data--2
31
Even if most respondents appear to be satisfied (or not), useful information can still be extracted by using the variation in responses, rather than the strict meaning of the responses But the survey design must allow for that, for example through:
Multiple choice responses – variation in response more likely to occur when surveys phrase satisfaction-related questions with multiple options, as opposed to a simple “yes/no” or “satisfied/dissatisfied”
Being specific – More variation in responses likely if questions are specific to different aspects or features of a school or health facility, as opposed to a single “are you satisfied with school/health center” type question
Implications for education and health services in Indonesia--1
32
Which aspects of health and education services in Indonesia matter the most for user satisfaction and how these differ across richer and poorer districtsInfrastructure
Quality of personnel and inputs
Participation in decision-making, governance and accountability
Implications for education and health services in Indonesia--2
33
InfrastructureSatisfaction with health facilities related more
with access to ancillary facilities supporting the main puskesmas, rather than the physical infrastructure of the puskesmas
But for schools in poor areas, infrastructure appears to be high priority among parents of students
34
Quality of personnel and inputsQuality of human resource and medicinal inputs
in health facilities is a major concern among users in poor districts
Indicators of teacher quality and student performance do not seem to matter much for user satisfactionDoes not necessarily imply that households are
ambivalent about education quality– rather that these indicators may not reflect the aspects of “quality” households care most about
Implications for education and health services in Indonesia--3
35
Participation in decision-making, governance and accountability Greater degree of community participation in decision-making for
facilities and better responsiveness of service providers to complaints appear to improve satisfaction
In education, satisfaction positively correlated with the extent of implementation of BOS (decentralization of service – e.g. school-based management, allocating funds to schools, participatory planning)
Indicators of participation and decentralized service delivery may proxy the “quality” valued by users; may also reflect a special value users may attach to being involved in the management of the facilities
Key questions: why these indicators matter for satisfaction, what explains the variations between rich and poor areas, and what that implies for the priorities of a government ?
Implications for education and health services in Indonesia--3
Thank you
Simple heuristic model of household satisfaction OLS of S on key characteristics of households and communities,
using the full household sample, separately for health and education Household variables include education, gender, age, ethnicity,
religion; district level variables such as urban/rural and rich/poor districts
Restricted to characteristics that service providers have little control over; variables related to objective quality of services omitted
Certain hhold characteristics have statistically significant correlation with reported satisfaction E.g. gender of hhold head, religion, ethnicity, education level,
household composition Rural respondents less likely to be satisfied with public facilities
they frequent Information on governance seems to matter
(i) households less likely to be satisfied when they know about complaints, but more when there was a follow-up in response; (ii) information about corruption/bribery in education strongly associated with lower satisfaction, but not so for health services
But these types of information highly likely to be endogenous 37
Sample Selection Strategy 1 Kabupaten/Kota (district)
Sampling frame: 434 districts (–) following 26 districts =408 districts
(21 (Aceh) – 2 (Nias) – 3 (pre-test locations: Kabu- Maros (South Sulawesi), Kabu- Pontianak (West Kalimantan), and Kabu- Muaro Jambi (Jambi))
Procedure1. 89 districts are selected randomly using SRS2. Rest of the districts are top-offs from ILGR districts,
USDRP, SPADA, and SPADA-Justice (WBOJ); 3. And 2 training districts : Kota Salatiga and Kabu- Boyolali
(Central Java) Total districts selected = 135 (with 1 dropped)
Sample selection strategy 2
Kecamatan (sub-district)Sampling frame: all kecamatans within respected districts (data
source: PODES 2005) Procedure1. Random districts: 3 sub-districts using PPS with # HH as weight2. Total sub-districts selected = 134 * 3 = 402 Desa/Kelurahan (village)Sampling frame: all villages within respected kecamatans with 50+
HHs; 2. 16 households within the village Procedure: 1. Pick 2 villages using PPS Total villages selected = 402 * 2 = 804
Sample selection strategy 3
Dusun (hamlet) Sampling frame: all hamlets within respected village Procedure: sort hamlet names alphabetically; select 2
hamlets from the sorted list: (a) the first and (b) the middle of the list
Total hamlets selected = 804 * 2 = 1,608 Household Get the most recent household list from head of hamlet.
Randomly select 8 households from the list. Total household = 1,608 * 8 = 12,864
Note: We use 88 randomly selected districts
Sampling criteria: Schools & Puskesmases
Sub District (3)
District Education Unit=1
SD-1: Junior High (1) Puskesmas (2)
SD-2: Junior High (1) Puskesmas (2)
SD-3: Junior High (1) Puskesmas (2)
Village(6)
V-1: Primary (1) Pvt.Paramed (3)Pvt. Doctor (1)
V-2: Primary (1) Pvt. Paramedic (3)Pvt. Doctor (1)
V-3: Primary (1) Pvt. Paramed (3) Pvt. Doctor (1)
V-4: Primary (1) Pvt.Paramed (3) Pvt. Doctor (1)
V-5: Primary (1) Pvt. Paramedic (3)Pvt. Doctor (1)
V-6: Primary (1)Pvt. Paramed (3)Pvt. Doctor (1)
Selection of facility respondentsType of respondents Total respondents per district
Head of Education facilities 1
Head of Health facilities 1
Primary school
Principals 6
Secondary data 6
School Committees 6
Junior High school
Principals 3
Secondary data 3
School Committees 3
Health
Head of Puskesmas 6
Puskesmas secondary data 6
Top Related