Download - THE SURPRISING LACK OF EVIDENCE ABOUT …home.hiroshima-u.ac.jp/cice/wp-content/uploads/2014/11/...OUTLINE 1.The Global Learning Crisis 2.Quiz: What Improves Learning? 3.Why Learning

THESURPRISINGLACKOF

EVIDENCEABOUTWHATWORKSINEDUCATION‐ IMPLICATIONSFORPOLICYMAKERS

NICHOLASBURNETT12NOVEMBER2014

JICATRAINING,HIROSHIMAUNIVERSITY

OUTLINE

1. TheGlobalLearningCrisis2. Quiz:WhatImprovesLearning?3. WhyLearningMatters‐ EvidenceonLearningandEconomicDevelopment4. TypesofEvidence5. Video:WhatWorksinPrimaryEducation?6. SourcesofEvidence7. WhyEducationInnovationMatters8. HowtoAssessInnovation?9. ImplicationsforPolicyMakers

2

1.THEGLOBALLEARNINGCRISIS

3Source:GMR2013

4


• Learning(e.g.UWEZOinEastAfrica)

TestpassratesforchildrenenrolledinStandard3

Notes:“combined”referstopassesonbothnumeracytestandatleastoneoftheliteracytestinthesurvey

TestpassratesforchildrenenrolledinStandard7

Source:UWEZO,2014

1. WeneedtofocusonWHATWORKSFORLEARNING

2. WeneedtoencourageINNOVATIONFORLEARNING

5


• ImplicationsoftheLearningCrisis

2.QUIZ:WHATIMPROVESLEARNING?

6

1. Increasedspendingoneducation

2. Mathematicsmaterialsatprimarylevel

3. Increasedteachers’salaries

4. Schoolbasedmanagement

5. Separatetoiletsforgirls

6. Extracurricularclubsforgirls

7. Lowcostprivateschoolsforthepoor

8. Conditionalcashtransferslinkedtogrades

Whichofthefollowingimproveslearning?

NO

YES

NOTCLEAR

NOTCLEAR

NO

YES

SOMETIMES YES

7

3.EVIDENCEONLEARNINGANDECONOMICDEVELOPMENT

SourceHanushek 2013

• EconomicDevelopment


8

• Individualreturnstoeducationarelinkedto

learning,notjusttoyearsofeducation

•Theoveralleconomybenefitsfromquality

education

• It’saboutchildrenlearning...


9

• Returnstocognitiveskills(literacy)aregenerallystrongacrosscountries


10

• EstimatedReturnstoCognitiveSkills:Examples

ThreestudiesconductedintheU.S.showa12%increaseinearningsforevery1SDincreaseinmathscores.(Mulligan,1999;Murnane etal,2000;Lazear,2003)

Chileandatashowsthat1SDincreaseintestscoresontheIALSisassociatedwithhigherearningsof15‐20%.(Sakellariou,2006)


11

• EducationQualityandEconomicGrowth

SourceHanushek andWoessman,2007


12


13

• A‘general’theoryofwhatinfluenceslearningoutcomes

Demand Supply

StudentsParents

Community

TeachersSchools

Education authoritiesCommunity

Endowments & Behaviors

of Policy actions, specific programs

Systemic reforms

Student learning

EducationPolicy

Economic, social, political context

Endowments & Behaviors

of

Source:VegasandPetrow2008

4.TYPESOFEVIDENCE

14

• RandomizedControlTrials – thegoldstandardforimpactevaluation• Useofcounterfactual• Advantages‐ reliable• Drawbacks– costs,findingsmaynotbepolicyrelevant• Difficulty– definingresults

• RegressionAnalysis• Standardstatisticalandeconometrictechnique• Advantages– simple,cheap• Drawbacks– selectionbias,correlationmaynotbecausation• Difficulty– definingresults

• AggregatingandInferring• Crowdsourcing‐ wisdomofthesuccessful?• Meta‐analysis• IdentificationofEssentialIngredients

• NoneofthesesayanythingaboutCostsorPovertyImpact• Needbenefit‐costanalysis,benefit‐incidenceanalysisetc.aswell

4.TYPESOFEVIDENCE

15

Whatsortofdatameasurelearning?

• Gradecompletion– allstudents?

• Exams– allstudents?

• Assessments– validstatisticalsamples• Nationalassessments– doesyourcountryhave?• Internationalassessments– EGRA,PISA,TIMSS,PIRLS,

SACMEQ,PASECetc.

• School‐basedorpopulation‐based– studentsorschool‐agepopulation?

5.VIDEO

16

https://www.youtube.com/watch?v=kX1Hp7ZkjO4

ShariKrishnaratne“whatworksineducationindevelopingcountries”

17

6.SOURCESOFEVIDENCE

• PrimaryEducationKrishnaratne etal.(2013)Qualityeducationforallchildren?Whatworksineducationindevelopingcountries.WorkingPaper20.3ie.http://www.3ieimpact.org/evaluation/working‐papers/working‐paper‐20/

• SecondaryEducationBanerjeeetal(2013)ExpandingAccessandIncreasingStudentLearninginPost‐PrimaryEducationinDevelopingCountries:AReviewoftheEvidence.AbdulLatif Jameel PovertyActionLab(J‐PAL)Post‐PrimaryEducationInitiativeReviewPaper2http://www.povertyactionlab.org/publication/expanding‐access‐and‐increasing‐student‐learning‐post‐primary‐education‐developing‐count

• Girls'EducationUnterhalter etal(2014).Girls’educationandgenderequality.EducationRigorousLiteratureReview2014.EPPI‐Centerhttps://www.gov.uk/government/uploads/system/uploads/attachment_data/file/326205/Girls__Education_Literature_Review_2014_Unterhalter.pdf

• LowCostPrivateSchoolAshleyetal(2014)Theroleandimpactofprivateschoolsindevelopingcountries.EducationRigorousLiteratureReview2014.EPPI‐Center:https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/307032/Private‐schools‐2014.pdf

• InnovationsEducationInnovationFoundation.http://www.educationinnovation.org/

• EarlyChildhoodDevelopmentEngleetal (2011).ChildDevelopment2:Chapter2Strategiesforreducinginequalitiesandimprovingdevelopmentaloutcomesforyoungchildreninlow‐incomeandmiddle‐incomecountries.Lancet.378:1339‐53

Behrmanetal(2014)Chapter3:PreschoolPrograminDevelopingcountries.InP.Glewwe (Eds.).EducationPolicyinDevelopingcountries,Chicago:UniversityofChicagoPress

References(seeannex)

18

• PrimaryEducation

Source:3ie2013

6.SOURCESOFEVIDENCE

Outcome Whatworks? Whatdoesn’t work? What promising?Notpromisingbutcouldlookfurther

researchWhatwedon’tknow

Enrolment/Attendance CCTsHealthinterventions(attendance)

Materials(attendance)

Teacher Resources(attendances)BuildingsECD(enrolment)SchoolFeeding(enrolment)

SBMSchoolFeeding(attendance)

Teacher Resources(enrolment)Health(enrolment)ECD(attendance)Materials(enrolment)VouchersSchoolsFeesProvidingInformation

Progression, Repetition,Dropout

Teacher Resources(dropout)CCTsSBM(progression)

SBM(dropout) Teacher ResourcesHealthECDMaterialsSchoolFeedingVouchersSchoolFeesProvidingInformation

TestScore Materials(maths)

Materials(language) Teacher ResourcesSBM(maths)SchoolFeeding(maths)

Teacher Resources(global)Health(maths,language)CCTs(global)SchoolFeeding(language)

Health(global)CCTsECDMaterials (global)SBMSchoolFeedingVouchersSchoolFeesProvidingInformation

19

• Girls’Education(ImpactonLearning)

Source:EPPI‐Center2014

6.SOURCESOFEVIDENCE

Interventions Strong Promising More needed

ResourceandInfrastructure

Resource Interventionincashforfamilyorchildrenimpactonsuccessinthegradesachieved

Theprovisionofadditional schoolsinunderservedareaimpactsongirls’learningoutcomes

Research intotheimpactofresourcesandinfrastructuretendstofocusonattendanceatschools.Moreevidenceinneededtoinvestigatetheimpactoftheseinterventionsonlearning

Institutional andPolicy

Grouplearning isassociatedwithimprovedlearningoutcomes

Learningoutsidetheclassroomsthroughformalandinformalextracurricularactivities(eg tutoringafter‐schoolclubs)hasapositiveeffectongirls’learningoutcomes

Thereispromisingevidencetoshow thatinformalteachertrainingtodevelopattitudesofinclusiontolerancehasapositiveimpact

Theinvolvementofwomeninschoolgovernanceandcommunitymobilizationandincommunityleadershipmayhaveapositiveimpactongirls’confidence

Girl‐friendlyschool,andquality–mix,orcombinedschool‐levelreformsmayhaveapositiveimpactongirls’learning

More researchisneededtoreinforceexistingpromisingevidencethattheempowermentofwomenteachersmayhaveapositiveimpactongirls’learning

Moreresearch isneededintotheimpactofschoolchoiceandinclusivestrategiesongirls’learning

NormandInclusion Thereis promisingevidencethatcomplementarylearningspacestoschool(e.g.NGOprogrammes,girls’clubs)areeffectiveingivingopportunitiestodiscussgenderequality

Teaching aboutpersonal,socialandhealthissuelinkedwithsexeducationbothatschoolandincomplementaryprogrammes iseffectiveinprovidingknowledge

Women’sliteracyprogrammes,particularlywhentheyofferwomenandgirlsthechancetodevelopliteracyknowledge,skills,andspacesforreflectivediscussion

Moreresearchneeded intoreligion,educationandshiftinggendernormsnotetheimportanceofkeyindividuals,thesignificanceofsupportivenetworkswithinfaithcommunities,andthepotentialforreligiousinstitutionstositedforcriticalreflectionongendernorm

Alsomoreneededontheeffectoftacklinggender‐basedviolenceinschools,workingwithboysongenderequality,worktoincludemarginalizedgirls,developingcombinedprogrmmes linkingworkinschoolsongenderequalitytocommunitywork

20

• LowCostPrivateSchools


6.SOURCESOFEVIDENCE

StrongEvidence ModerateEvidence

Teachingisbetterinprivate schoolthaninstateschools,intermofhigherlevelsofteacherpresenceandteachingactivityaswellasteachingapproachesthataremorelikelytoleadtoimprovedlearningoutcomes

Privateschoolpupils achievebetterlearningoutcomeswhencomparedwithstateschools.However,thereisambiguityaboutthesizeofthetrueprivateschooleffect.Inadditionmanychildrenmaynotbeachievingbasiccompetencieseveninprivateschools

Thecostofeducationdeliveryislowerinprivateschoolsthanstateschoolsoftenduetolowersalariesforprivateschoolteacherscomparewiththeirgovernmentschoolcounterparts.Moreoftheevidencedoesnorigorouslyanalyzethecost‐effectivenessofprivateschools;however,thereissomelimitedevidencetoindicatehighercost‐effectivenessofprivateschoolsthanstateschoolsinspecificcontexts

Girlsarelesslikelythanboystobeenrolledinprivateschools.However,thisfindingiscontextspecificwithaminorityofstudiesfindingthatprivateschoolsreducethegendergapincertaincontexts

Perceivedbetterqualityofprivateschools(intermsofteaching,teacherattendance,schoolperformance,smallclasssize,discipline)comparedwithstateschoolsisakeyfactorinparents’choiceofprivateschools.OtherimportantfactorscitedincludeEnglish‐languageinstruction,futureoccupationpossibilitiesandpromotionratestosecondaryschool.

Theperceptionof‘privateschoolsasbetterquality’isinformedinformally,oftenthroughparents’informalsocialnetworks;suchsourcesplayasignificantbutoftenunder‐recognisedroleininformingusersintheirchoiceofschool.

Attemptsbystatestointerveneintheprivateeducationsectorareconstrainedbyalackofcapacity,legitimacyandknowledgeofthesectortoimplementeffectivepolicyframeworks.

Wherestateregulationofprivateschoolsexists,itisnotnecessarilyeffectiveormaybeselectivelyenforcedofferingopportunitiesforrentseekingandbribery.Althoughthefindingsaremostlynegative,therearesomeexamplesofpositivestateregulationsupportingtheexpansionofprivateschoolprovision;howevertherearealsoconcernsthatprivatesectorprovisionmaybepromotedbystateswithoutadequateregulationandqualitycontrols.

21

• LowCostPrivateSchools


6.SOURCESOFEVIDENCE

Inconclusive Limited

Theevidenceisambiguousaboutwhetherprivateschoolsgeographicallyreachthepoor.Althoughprivateschoolsarecontinuingtofocusonurbanareas,theyarealsobecomingincreasinglyprevalentinruralareas;butresearchcautionsagainstassumingthismeanstheyarereachingthepoor.

Thereisasmallbutconsistentevidencebasethatprivateschoolsaremoreexpensivethanstateschoolsintermsofbothschoolfeesandhiddencostssuchasuniformsandbooks.

Theevidenceonwhetherthepoorareabletopayprivateschoolfeesisambiguous.Mostisneutral,someisnegative,butthereisnopositiveevidence.Anumberofstudiesfindthataverysmallminorityofchildrenoflowereconomicquintilesaccessprivateschools.Financialconstraintsareakeyfactorlimitingorpreventingpoorerhouseholdsfromenrollingtheirchildreninprivateschools.Wherechildrenofpoorerhouseholdsdoattendprivateschools,researchindicatesthatwelfaresacrificesaremadeandcontinuedattendanceisdifficulttosustain.

Thereareveryfewstudiesaddressingrigorouslytheaccountabilityofprivateschoolstousers.Ofthese,thereissomeconsistentevidencethatusersparticipateinandinfluencedecisionmakinginprivateschools.Whileasmallbodyofmainlyanecdotalevidenceindicatesthatteachersandschoolsmayrespondtoparents’demandsandcomplaintsandultimatelythepotentialthreatofparentsexercising choice,noevidencewasfoundinthestudiesreviewedthatusersdoinfactchangeschoolsinresponsetoqualityconcerns,oraremorelikelytodosointhecaseofprivateschoolsthangovernmentones.

Thereislimitedevidencetosupportanyfirmconclusionaboutthefinancialstabilityofprivateschools.However,theevidenceavailableindicatesthatprivateschools,especiallyLFPs,maybevulnerabletoclosingdownaftershortperiodsoftime.Howevertherewasalsocounterevidenceofestablishedrecognizedprivate schoolsoperatingovermanyyears.Importantlyanassessmentofthesustainabilityofdifferentfinancialmodelsislackingintheliterature.

Theevidenceonsubsidiesislimitedinscope,sizeandcontext,butonedonorfundedprogramme inPakistanindicatesthatconditionalandtargetedsubsidiescanraisethequalityofinputs.

Thereislittleevidencetosupportorrefutethequestionofthesystem‐wideeffectsofprivateeducation.Theevidencebaseonwhetherprivateschoolscomplementorcompetewithgovernmentschoolprovisionisverysmall,howeversomeevidenceindicatesasupply‐sidesynergybetweengovernmentandprivateschoolsprovisionandthereisevidencethatprivateschoolsarefillinggapswheresupplyofgovernmentschoolsislow,butalsowheregovernmentschoolsareperformingpoorly.Theevidenceonwhethertheeffectofcompetitionistodriveupthequalityofpublicschoolsortodepleteitbyencouragingmoreablestudentstoexitthestatesectorissparseandcontested.

7.WHYEDUCATIONINNOVATIONMATTERS

22

Source:UNESCOGMRPolicyPaperNo.14,2014

Numberofout‐of‐schooladolescentsoflowersecondaryschoolagebyregionandsex,2000‐2012

Numberofout‐of‐schoolchildrenofprimaryschoolagebyregionandsex,2000‐2012

• OutofSchoolChildren


23Source:GMR2013

• LearningCrisis


24Source:GMR2013

• Effectiveness


25

Programs,PoliciesandPractices

thatincreaseaccesstoorimprove

qualityorreducecostsof

educationforthepoor

• WhatareEducationInnovations?


26

1Progressineducationdoesnotgrowandspreadfastenough

2Solutionsdoexist– promisinginnovations ineducationareconstantlyemerging

3…BUTtheiruptakeanddiffusionisslow!

• EducationChallenge:TheReality&TheOpportunity


27

Blueprint Validate Prepare Scale

Developideaandcompellingvalueproposition

Testandrefinetheinnovation.Mayinvolvesomegrowth.

Enhancetheconditionsrequiredforscaling(i.e.,customerawareness,

effectivesupplychains)

Rolloutmodeltoreachalargenumberofcustomers

Takes5‐10yearstogothroughfirstthreestages

Source:BlueprinttoScale,producedbyMonitorGroupincollaborationwithAcumenFund

• UnderstandingtheEvolutionofanInnovation:FourStages

GlobalPublicGoodPlatformforDataandConnectionseducationinnovations.org


28

1

23

Largestglobalpublicdatabaseofinnovations

Newanalysisabout‘whatworks’,howandwhy

Connectionsleadingtoimprovement,scale‐upandadaptation

• TheR4DCenterforEducationInnovations

8.HOWTOASSESSINNOVATION?

29

Interpretations Requirements Expand

Samecontext

But5to10verydifferentfrom50to10,000

Replicate

Differentcontext

Adapt

Samecontext,or

Differentcontext

Analyticalrequirements

Clearidentificationandanalytic

separationof“activeingredients”and

ofcontextualfactors.

Assessmentoftheeffectivenessofthe

activeingredients

Howtodothis?Evolvingtechniques….

• ScalingUp‐ Looseterm……..needtobeprecisewhenusing

8.HOWTOASSESSINNOVATION?

30

Theactivitiesbelowrepresentkeycheckpointsintheprocessofidentifyingpromisingglobalmodelspotentiallyadaptabletoothercontexts.Theorderofthestagesmayvarybyuser,desiredoutput,andinformation/resourcesavailable.Asaniterativeprocess,usersmayrevisitcertainstepstofurtherrefineanalysis.

Searchforprogrammodels

Identifytheactiveingredient(s)

Determineeffectiveness

Assessscalability

Basedontheproblemyouaretryingtosolveforandspecifiedparameters,whatmodelscurrentlyexist?

Whatarethecorecomponentsorattributesoftheprogramthatmakeitsuccessful?Whatarethecontextualfactorsfortheactiveingredient?

Whatarethebarriersandleverstoscalingtheactiveingredient(s)inaspecificcontext?

Whatevidenceexistsontheeffectivenessofidentifiedactiveingredient(s)?

OUTPUTSShortlistofprogrammodels Listofisolatedactiveingredients Narrowedlistofactiveingredients Listofbarriersandleversto

adaptation

• ProposedScalingFramework

8.HOWTOASSESSINNOVATION

• R4DCenterforEducationInnovations(CEI):http://www.educationinnovations.org/• Focusonthepoorinlowandmiddleincomecountries• Over500documentedprograms,mainlynon‐state

• WorldInnovationSummitonEducationHub:http://www.wise‐qatar.org/edhub• Over100describedinnovationsinrichandpoorcountries

• OECDCenterforEducationResearchandInnovation:http://www.oecd.org/edu/ceri/• FocusonOECDcountries

• STIR:http://www.stireducation.org/• Focusonteacherinnovations,IndiaandnowexpandingtoUganda

31

• SourcesofInformationaboutInnovation

9.IMPLICATIONSFORPOLICYMAKERS

32

1. Makingpolicywithinsufficientinformation

• Donotassumeyouknow,e.g.“”wehavealwaysdoneitthisway”

• Donotassumewhatspecialinterestssay,e.g.teacherunionorbankslendingfor

highereducation

• Constantlyscanforrelevantinnovations

• Applylogframe ortestabletheoryofchange

• Seewhichelementsmayhaveevidence

• Forthosewhichdonothaveevidence,makereasonableassumptionsbutinstall

M&Etoverifyornottheseassumptions

• Anotherapproach:rewardresultsbutdon’ttellimplementershowtoachievethem• Results‐basedfinancing• Output‐basedaid• Payforperformanceetc.

9.IMPLICATIONSFORPOLICYMAKERS

33

2. Buildingevidence‐gatheringintopolicy

• Allocatefundsforresearchandmanageresearchbudgetsactively

• Insistexternalfinanceforresearchflowthroughtheministry

• Setuppolicyadvisorycommitteesthatincluderesearchers

• Keepafocusonoutcomesand,toalesserextent,outputs

• Learnfromothercountries’experience

• AlwaysputM&Einplacewhenscalinginnovationsandnewpolicies

34

CONTACTINFO:[email protected]

Thankyou