Education PPP: an Output-Oriented Approach
description
Transcript of Education PPP: an Output-Oriented Approach
![Page 1: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/1.jpg)
www.epdc.org
Education PPP: an Output-Oriented Approach
No schoolingPrimary
SecondaryVocational
Tertiary
Exp = PPP ×Quantity ×Quality
1
![Page 2: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/2.jpg)
www.epdc.org
Outline: 1. Quantity 2. Quality 3. Conceptual model
No schoolingPrimary
SecondaryVocational
Tertiary
2
![Page 3: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/3.jpg)
www.epdc.org
Measuring the Quantity of Education
• Basic element is pupil counts: how many pupils consumed education services?
• Enrollment figures generally available from national Education Management Information Systems (EMIS), but must be validated
• Actual services consumed by pupils calculated using a time-on-task measure of the pupil-hours volume of instructional activity
3
![Page 4: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/4.jpg)
www.epdc.org
Validating enrollment figures (method 1)
• Adjust for school coverage (are all schools included in National EMIS systems?)
• Sample survey of schools• Compare to time series trends• Adjust for pupil expenditures• Ensure all sub-sectors report and look for
consistencies
4
![Page 5: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/5.jpg)
www.epdc.org
Validating Enrollment Figures (method 2)
• Compare Enrollments from EMIS and Household Surveys, such as DHS
• Differentiating between Enrollment and Attendance (EMIS captures enrollment, household surveys – attendance)
• Use Verification Protocols developed by AED, UIS, UNICEF and UNESCO
5
![Page 6: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/6.jpg)
www.epdc.org
Method 2 verification protocols
• Use UIS data when no survey MICS/DHS data are available
• Use MICS/DHS when no UIS data available• Use UIS when data is < 5% difference,• Use expert when data is > 5 % difference
6
![Page 7: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/7.jpg)
www.epdc.org
Adjust Quantity by Estimating Pupil Hours
• Subtract Days Lost from Official School Year• Collect Teacher Absence Data and Roll-up• Collect Student Absence Data and Roll-up• Calculate Daily Time Loss• Survey Time on Task • Sample Schools and Estimate Absence Data
across Stages/Levels/Organization
7
![Page 8: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/8.jpg)
www.epdc.org
Stratification
• Disaggregation by ISCED levels (some national education systems do not align neatly w ISCED)
• Possibly, disaggregation by:– Public– Private– Complementary models (religious, community
schools)
8
![Page 9: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/9.jpg)
www.epdc.org
- Moving towards a common learning metric - Composite Learning Scores in Education PPP- Data imputation will help address the
problem of missing quality scores
Measuring the Quality of Education
9
![Page 10: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/10.jpg)
www.epdc.org
Why impute learning scores in education PPP?
• A fair assessment of value is inevitably linked to outputs• In education, outputs are learning outcomes, measured
by assessments of student achievement• Participation in assessments is growing, but far from
global• Imputation of missing values is a way of accounting for
quality in a broader group of countries than those currently available
• International and regional assessments will be used.
10
![Page 11: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/11.jpg)
www.epdc.org11
East Asia & Pacific
Europe & Central Asia
Latin America & Caribbean
Middle East & North Africa
South Asia
Sub-Saharan Africa
East Asia & Pacific
Europe & Central Asia
North America
East Asia & Pacific
Europe & Central Asia
Latin America & Caribbean
Middle East & North Africa
Deve
lopi
ng o
r tra
nsiti
onal
O
ECD
Non-
OEC
D Hi
gh In
com
e
0 5 10 15 20 25
6
14
5
10
0
4
4
20
2
2
3
0
8
2
10
3
2
0
1
1
17
2
2
2
1
3
3
12
7
2
0
0
4
21
2
2
3
0
2
PISA
PIRLS
TIMSS
International Achievement Studies: Regional Coverage
![Page 12: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/12.jpg)
www.epdc.org
13
20
15
0 10 20 30
SACMEQ (Sub-Saharan Africa)
PASEC (Sub-Saharan Africa)
LLECE (Latin America & the Caribbean)
Number of Countries
Additional Sources of Data: Regional Studies
12
![Page 13: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/13.jpg)
www.epdc.org
90
19
74
Availability of Student Achievement Scores: ALL ICP PARCIPATING COUNTRIES (N=182)
International as-sessments (TIMSS, PIRLS, PISA)Regional Only (PASEC, SACMEQ, LLECE)No test information
13
![Page 14: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/14.jpg)
www.epdc.org
Towards a Common Quality Adjustment
• Data imputation is necessary to include all countries in a quality adjustment process
• All available information must be used• Reliability and validity will be tested
14
![Page 15: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/15.jpg)
www.epdc.org
Imputation Methods: Some Examples• Crouch (Crouch and Vinjevold, 2006)
– Recursive regression imputation in several steps, based on correlation with TIMSS 1999 scores
– Univariate regressions with target learning score (TIMSS 1999) as dependent variable, one assessment at a time as sole predictor
– Result: several predicted scores for some countries, weighted average taken as imputed value
– Weights based on correlation with TIMSS 1999• Hanushek and Woessmann (2009, 2010)
– “Calibration” of major international achievement studies into a new scale.
– Based on U.S. performance in international studies, and NAEP as reference point.
– Mean and standard deviation of 13 OECD countries
15
![Page 16: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/16.jpg)
www.epdc.org
Multiple Imputation (MI)• Long-established methodology for treating missing
data (for overview see Rubin, 1996; Allison, 2002) • Allows to use all available information• Accounts for uncertainty associated with imputation
(multiple estimates generated)• Imputes more than one variable with missing data at
a time
• Drawback: demands on technology/ time to produce reliable estimates.
16
![Page 17: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/17.jpg)
www.epdc.org
• Define dataset: three “core” assessments (state-of-the-art) and predictors of education quality
– GDP per capita– Age structure (percent youth)– Per pupil expenditure– Teacher-student ratio– Other variables known or theorized to predict quality
• Missing values on predictor variables will also be imputed
Imputation of Learning Scores: the Process
17
![Page 18: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/18.jpg)
www.epdc.org
Multiple Imputation (MI): the Mechanics• Conditional prediction models are specified for each
of the variables with missing data• MI runs iteration cycles “filling in the blanks” in all of
the variables across 3-5 chains, based on the specified models
• The goal is to achieve convergence of the distributions of imputed values across the chains, thereby increasing reliability
• Estimates are then pooled across the chains• The result is 3-5 completed (imputed) datasets
18
![Page 19: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/19.jpg)
www.epdc.org
The Composite Learning Score: Stages• Because of the high proportion of missing data on learning
assessments, the construction of the learning scores will be done in stages
• Stage 1: Countries with international assessments: Impute all three “core” assessments, construct single score from weighted average (90 out of 183)
• Stage 2: Countries with regional assessments: impute one composite learning score (19 countries)
• Stage 3: Countries with no assessments, but complete data on predictors of quality: impute one composite learning score (TBD: total is 74)
• Actual or imputed learning score will be transformed to a specified scale (currently Mean= 500; SD=100)
19
![Page 20: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/20.jpg)
www.epdc.org
Sources of Error and Threats to Validity• Unequal, nonrandom geographic representation in
international assessments• Regional assessments and EGRA lacking in standardization,
hence lower reliability• Large proportion of ICP countries (74) never participated in
learning assessments, hence quality scores will be derived solely based on inputs and proxies.
• MI will account for some of the uncertainty (averaging out across multiple estimates)
• Crude methods (e.g. single imputation using regression or predictive mean matching) may be applied for difficult cases.
20
![Page 21: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/21.jpg)
www.epdc.org
After the Imputation: Adjustments
• Once in the PPP model, learning scores will be adjusted for student SES and length of schooling
• SES Index: variables from available international achievement databases, as well as household surveys
• Length of schooling: weighted average of years spent through graduation from secondary, based on the proportion attending preschool
21
![Page 22: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/22.jpg)
www.epdc.org
After the Imputation: Adjustments
• Once in the PPP model, learning scores will be adjusted for student SES and length of schooling
• SES Index: variables from available international achievement databases, as well as household surveys
• Length of schooling: weighted average of years spent through graduation from secondary, based on the proportion attending preschool
22
![Page 23: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/23.jpg)
www.epdc.org 23
Value ofEducationServices
Quantity
Quality
Conceptual model
![Page 24: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/24.jpg)
www.epdc.org 24
Value ofEducationServices
Volume
Quality
# PupilsVerification,Adjustments (Moses, Stukel)
EMIS counts
Other sources
V = f(Pupils, y)Pupils = EMIS counts x adjustmentORV = HH survey attendance
![Page 25: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/25.jpg)
www.epdc.org 25
Value ofEducationServices
Time on Task Volume
Quality
# PupilsVerification,Adjustments (Moses, Stukel)
EMIS counts
Other sources
![Page 26: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/26.jpg)
www.epdc.org 26
Value ofEducationServicesiScore
(composite learning score)
International assessments
Time on Task Volume
Quality
# PupilsVerification,Adjustments (Moses, Stukel)
EMIS counts
Other sources
Predictors ofquality
![Page 27: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/27.jpg)
www.epdc.org 27
Value ofEducationServicesiScore
(Crouch, MI)
International assessments
SES adjustment
Time on TaskVolume
Quality
# PupilsVerification,Adjustments (Moses, Stukel)
EMIS counts
Other sources
HH surveys
GDP, other
![Page 28: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/28.jpg)
www.epdc.org 28
Average years to 8th grade
Preschool duration & coverage
Repetition
Dropout
Value ofEducationServicesiScore
(Crouch, MI)
International assessments
SES adjustment
Time on TaskVolume
Quality
# PupilsVerification,Adjustments (Moses, Stukel)
EMIS counts
Other sources
HH surveys
GDP, other
![Page 29: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/29.jpg)
www.epdc.org 29
Average years to 8th grade
Preschool duration & coverage
Repetition
Dropout
Value ofEducationServicesiScore
(Crouch, MI)
International assessments
SES adjustment
Time on TaskVolume
Quality
# PupilsVerification,Adjustments (Moses, Stukel)
EMIS counts
Other sources
HH surveys
GDP, other
Earnings ratioTertiary/ Lower secondary
Tertiary adjustment
![Page 30: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/30.jpg)
www.epdc.org
Further notes
30
All calculations by ISCED levelPrivate, public, religious may be aggregatedQ applied to preschool – secondaryQ(t) separate for tertiaryData on key inputs available for most countries
Pilot to include: EMIS count verificationsTime on task estimatesLearning assessments (?)
![Page 31: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/31.jpg)
www.epdc.org
The Roadmap
31
1. (ICP/Regional and External) Refine proposed conceptual model of output methodology into an implementable instrument
2. (External) First-level secondary data assessment and collection (for model-testing phase)
3. (External) iScore construction: imputation, adjustment.4. (External) Identify insufficient data coverage on
important predictors, develop plan to collect missing data.
![Page 32: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/32.jpg)
www.epdc.org
The Roadmap, cont’d5. (Local/regional and external) Train local or regional technical
teams to verify pupil counts, implement the sample survey for EMIS information verification.
6. (Local/regional and external) Train local or regional technical teams to assess actual instruction time, implement survey to collect missing information on instruction time.
7. (Local/regional) Assess whether EMIS counts or household survey estimates are the more reliable source of pupil count information; verify EMIS counts
8. (Local/regional) Implement assessment and surveys of actual instruction time
32
![Page 33: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/33.jpg)
www.epdc.org
9. (External, but can train regional experts) Collect SES, repetition, dropout, and preschool attendance data for all ICP countries where such information is available.
10. (External, but can train regional experts) Calculate Quality and Quantity for all countries with complete dataset.
11. (External) Rerun the imputation models with newly collected information, in particular, time-on-task.
33
The Roadmap, cont’d
![Page 34: Education PPP: an Output-Oriented Approach](https://reader035.fdocuments.us/reader035/viewer/2022062315/56816571550346895dd8088b/html5/thumbnails/34.jpg)
www.epdc.org
12. (External) Evaluate the value added of time of task, and the difference from validated pupil enrollment numbers.
13. Analyze and process the information into PPP adjustment values for the piloted countries, to evaluate the process, and to propose adjustments where necessary for a full-scale implementation of the output approach to education PPPs.
34
The Roadmap, cont’d