Impact Evaluation of the School Electrification and ... · PDF fileFaustino & Medina...
Transcript of Impact Evaluation of the School Electrification and ... · PDF fileFaustino & Medina...
Impact Evaluation of the School Electrification and Education (SEEd) Project of the Alliance for Mindanao Off grid Renewable Energy
Program (AMORE): A Utilization of Propensity Score Matching (PSM)
Nelson C. Faustino, PhD, Former M&E Specialist, SEEd-AMORE
Michael Arieh P. Medina, PhD, Faculty, CFES, Central Mindanao University
Presented at the “Making IE Matters Conference” ADB Headquarters, Metro Manila, Philippines September 3-5, 2014
Outline
• Background of the Study
• Objectives
• Methods
• Results and Discussions
• Conclusions
• Recommendations
Faustino & Medina #iematters 2
Background of the Study
• Out of ~5,600 secondary schools:
– 67% of schools scored within the “lower average” range in the National Achievement Test (NAT)
– ~1/3 made it to the upper score ranges
– Only ~1% were rated superior
• Source: DepEd Order No. 72 s. 2011
Faustino & Medina #iematters 3
Background of the Study
• The Philippines ranked 5th out of 8 ASEAN countries
• Last in rank in terms of quality of Math and Science education and with the capacity for innovation.
– Global Competitiveness Report, WEF (2011)
Faustino & Medina #iematters 4 https://www.youtube.com/watch?v=z4bEkzwc0w4
Background of the Study
• …MULTIMEDIA can improve learning and retention of material presented during a class session or individual study period, as compared to “traditional” lectures or study materials…
– Ludwig et al. (2004)
Faustino & Medina #iematters 5
The SEEd-AMORE Program
• USAID-funded in collaboration with SunPower Foundation, Intel Phils., and DepEd.
• Access to RE (solar) by off grid rural schools
• Provision of multimedia educational materials
– Improving English Language Teaching and Learning in Mindanao Program (Real World Productions)
– e-Media Program (ABS-CBN Foundation)
– Knowledge Channel
Faustino & Medina #iematters 6
Objectives of the Evaluation
• Determine the influence of the SEEd-AMORE program on the educational performance of student beneficiaries (NAT Score).
– Compare the NAT Scores of student beneficiaries and non-beneficiary students
– Determine the degree of influence of the extent of ETV utilization on the NAT Scores of student beneficiaries.
Faustino & Medina #iematters 7
Methods
• Survey of 409 project beneficiaries and 534 non-beneficiaries.
• Propensity Score Matching (PSM)
• t-test for independent groups
• Multiple Linear Regression (MLR)
Faustino & Medina #iematters 8
Results and Discussions
• Unmatched Sample:
• Beneficiaries = 409, non-beneficiaries = 534
NAT Score Beneficiaries Non-Beneficiaries
Filipino 62.57 60.81
Math 67.85 62.65
English 62.50 61.56
Science 63.74 64.34
HeKaSi 66.80 63.11
MPS 64.69 62.49 Faustino & Medina #iematters 9
Results and Discussions
• Unmatched Sample:
• Beneficiaries = 409, non-beneficiaries = 534
NAT Score t-value p-value
Filipino 1.476 0.140
Math 3.187 0.001
English 0.678 0.498
Science 0.487 0.626
HeKaSi 2.811 0.005
MPS 1.820 0.069 Faustino & Medina #iematters 10
Results and Discussions
• propensity score matching is a methodology used in attempting to provide unbiased estimation of treatment-effects.
– Rosenbaum & Rubin (1983)
Faustino & Medina #iematters 11
Results and Discussions
• Step 1. Binary Logistic Regression Analysis
– In order to identify significant sources of bias
Covariates B Wald p-value Exp(B)
4 Ps Program (Non-Beneficiary) -0.565 14.850 0.000 0.569
Ownership of House and Lot -0.105 4.299 0.038 0.900
No. of Appliances at Home 0.310 24.514 0.000 0.733
Home Electricity Source (Off Grid) 0.398 4.486 0.034 1.488
Honors Received (No Honors) -0.463 10.187 0.001 0.629
α=0.05 Faustino & Medina #iematters 12
Description of the PSM (Nearest Neighbor) Methods
Method Replacement Caliper Match Ratio
1 With Without 1:1
2 Without Without 1:1
3 With 0.2 1:1
4 Without 0.2 1:1
5 With 0.2 1:2
6 Without 0.2 1:2
7 With Without 1:2
8 Without Without 1:2
Faustino & Medina #iematters 13
Results and Discussions
• Step 2. PS Matching (Nearest Neighbor)
8 Matching Methods
Sufficient Units for Methods
1,2,3,4,5 & 7
Not enough Control Units for Methods
6 & 8
Significant Imbalance
for Method 2
Most Improvement in Balance for
Method 4
PS Matching Overall Balance Multivariate Imbalance
Faustino & Medina #iematters 14
Results and Discussions
• Unmatched Sample:
• Beneficiaries = 409, non-beneficiaries = 534
NAT Score t-value p-value
Filipino 1.476 0.140
Math 3.187 0.001
English 0.678 0.498
Science 0.487 0.626
HeKaSi 2.811 0.005
MPS 1.820 0.069 Faustino & Medina #iematters 15
Results and Discussions
• Matched Sample (Method 4):
• Beneficiaries = 332, non-beneficiaries = 332
NAT Score t-value p-value
Filipino 3.054 0.002
Math 4.764 0.000
English 1.750 0.081
Science 1.752 0.080
HeKaSi 3.435 0.001
MPS 3.398 0.000 Faustino & Medina #iematters 16
Results and Discussions • Multiple Linear Regression Analysis
Dependent Variable = NAT MPS (N=409)
Factors B β t-value p-value
(Constant) -120.48 -5.1 0.000 No. of Major Appliances 3.323 0.275 6.318 0.000 Attendance (%) 0.445 0.191 4.455 0.000 Parents are 4P’s members 9.377 0.276 6.481 0.000 Average School Grade (%) 0.928 0.171 4.024 0.000 Nutritional Status 4.133 0.105 2.479 0.014 Sineskwela 9.651 0.091 2.172 0.030 Mathtinik 9.699 0.115 2.398 0.017 Bayani 7.148 0.115 2.335 0.020
Faustino & Medina #iematters 17
Conclusions
• The SEEd-AMORE program was able to influence the performance of student beneficiaries in terms of the NAT MPS
• The extent of utilization of multimedia learning materials (Sineskwela, MathTinik, and Bayani) influences the NAT MPS of student beneficiaries
• Other essential factors that influences school performance are: family income, attendance, 4P’s, School Grade, and Nutrition.
Faustino & Medina #iematters 18
Recommendations
• Replication of the SEEd-AMORE program in other geographical regions.
• Intensification of the utilization of multimedia learning materials in the basic education curricula.
• Technical assistance to teachers in adapting to technology.
• Further evaluation of other programs related to the improvement of student performance in rural schools (e.g. 4P’s, Nutrition, etc.)
Faustino & Medina #iematters 19
Literature Cited in this Presentation
Alliance for Mindanao Off Grid Renewable Energy (AMORE) Program (2011). School Electrification and Education (SEEd) Manual.
Tubeza, P. (2011). DepEd seeks higher test scores for RP students. Retrieved July 11, 2012 from http://newsinfo.inquirer.net/inquirerheadlines/nation/view/20100108-246133/DepEd-seeks-higher-test-scores-for-RP-students
Ludwig, T. et al. (2004). Using Multimedia In Classroom Presentations: Best Principles. Prepared for the Society for the Teaching of Psychology Pedagogical Innovations Task Force December 2004.
Rosenbaum, P. & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70 (1): 41–55. DOI:10.1093/biomet/70.1.41.
World Economic Forum (2011). Global Competitiveness Report
Faustino & Medina #iematters 20
Acknowledgement
• University of Southeastern Philippines (USeP), College of Governance and Business (CGB)
• Central Mindanao University (CMU), College of Forestry and Environmental Science (CFES)
• Rec E. Eguia, PhD (USeP-CGB)
• Rural Off Grid Schools in Mindanao
• United States Agency for International Development (USAID) – SEEd-AMORE Program
• ADB, 3ie, PIDS
Faustino & Medina #iematters 21
Daghang Salamat
Maraming Salamat
Thank You
Faustino & Medina #iematters 22
Contact Details: Website: http://gradschooljungle.blogspot.com Email: [email protected], [email protected], [email protected]