STEM Instruction Improvement Programs Improve Student … · response to calls to increase both...
Transcript of STEM Instruction Improvement Programs Improve Student … · response to calls to increase both...
![Page 1: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/1.jpg)
STEM Instruction Improvement Programs Improve Student Outcomes
How should teachers spend their STEM-focused professional learning time? To answer this question, we analyzed a recent wave of rigorous new studies of STEM instructional improvement programs. We found that programs work best when focused on building knowledge teachers can use during instruction: knowledge of the curriculum materials they will use, knowledge of content and how content can be represented for learners, and knowledge of how students learn that content. We argue that such learning opportunities improve teachers’ professional knowledge and skill, potentially by supporting teachers in making more informed in-the-moment instructional decisions.
Suggested citation: Hill, Heather, Kathleen Lynch, Kathryn Gonzalez, and Cynthia Pollard. (2019). STEM Instruction Improvement Programs Improve Student Outcomes. (EdWorkingPaper: 19-135). Retrieved from Annenberg Institute at Brown University: http://www.edworkingpapers.com/ai19-135
Heather C. HillHarvard University
Kathleen LynchBrown University
Kathryn E. GonzalezHarvard University
Cynthia PollardHarvard University
VERSION: September 2019
EdWorkingPaper No. 19-135
![Page 2: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/2.jpg)
STEMInstructionImprovementProgramsImproveStudentOutcomesHeatherC.Hill,KathleenLynch,KathrynE.Gonzalez,CynthiaPollard
SuggestedCitation:Hill,H.C.,Lynch,K.,Gonzalez,K,&Pollard,C.(forthcoming).STEMInstructionImprovementProgramsImproveStudentOutcomes.PhiDeltaKappan.
![Page 3: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/3.jpg)
AbstractHowshouldteachersspendtheirSTEM-focusedprofessionallearningtime?Toanswerthisquestion,weanalyzedarecentwaveofrigorousnewstudiesofSTEMinstructionalimprovementprograms.Wefoundthatprogramsworkbestwhenfocusedonbuildingknowledgeteacherscanuseduringinstruction:knowledgeofthecurriculummaterialstheywilluse,knowledgeofcontentandhowcontentcanberepresentedforlearners,andknowledgeofhowstudentslearnthatcontent.Wearguethatsuchlearningopportunitiesimproveteachers’professionalknowledgeandskill,potentiallybysupportingteachersinmakingmoreinformedin-the-momentinstructionaldecisions.
![Page 4: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/4.jpg)
HowshouldteachersspendtheirSTEM-focusedprofessionallearningtime?Recentnationalsurveydata(Baniloweretal.,2018)suggeststeachersofSTEM(science,technology,engineeringandmathematics)devotetheirprofessionallearningtimetostudyingstatestandards,analyzinginstructionalmaterials,deepeningtheirunderstandingofcontentandstudentthinkingaboutcontent,learningaboutassessment,andstudyingstudentdata.Whatwedon’tknowistheextenttowhichsuchactivitiesimprovestudents’academicoutcomes.InarecentsynthesisoftheresearchliteratureonSTEMinstructionalimprovement,wesetouttodeterminejustthat.Wefoundtheprogramsthatworkbesttendtofocusonbuildingknowledgeteacherscanuseduringinstruction:abouthowtousecurriculummaterials,aboutcontentandhowcontentcanberepresentedforlearners,andabouthowstudentslearnthatcontent.Wearguethatsuchlearningopportunitiesimproveteachers’professionalknowledgeandskill,potentiallybysupportingteachersinmakingmoreinformedin-the-momentinstructionaldecisions.AStudyofStudiesConductingsynthesesoftheliteratureoninstructionalimprovementprogramsisnotnew.Suchsynthesesgenerallyhavetwogoals:tocalculatetheaveragesizeofprogrameffectsonstudentoutcomes,andtolearnwhetherspecificprogramcharacteristicspredictvariationintheseeffects.However,upuntilrecently,attemptstoperformsynthesesusingonlyrigorousstudies–thosethatcompareparticipatingteacherstoasimilarsetofnon-participants,typicallyviarandomization–returnedtoofewstudiestoachievethelattergoal.Forinstance,Yoonandcolleagues(2007)couldfindonlynineELA,math,andsciencestudiesthatmetfederalWhatWorksClearinghouse(WWC)evidencestandards.InSTEM,Gerstenetal.(2014)locatedonlyfivemathematicsprofessionaldevelopmentstudiesthatmetWWCstandards.Whilethesenumberswouldallowthemeta-analysisauthorstocalculatetheoveralleffectoftheseprogramsonstudentoutcomes,neithercouldlinkspecificprogramfeaturestothoseoutcomes.However,followingcallsforstrongerresearchintotheimpactofeducationalinterventions(e.g.,Shavelson&Towne,2001),federalresearchportfoliosbeganintheearly2000stoprioritizeresearchmethodsthatallowforcausalinference,andtorequireimprovedstudentoutcomesasanindicatorofprogramsuccess.Dollars’andscholars’turntowardusingcausalmethodsandexaminingstudent-levelimpactshasresultedinawealthofnewstudies.Thesenewstudies,wereasoned,wouldpermitmoreextensiveempiricalanalysesthanpriorsyntheses.Withthisinmind,wesetouttoreviewstudiesonSTEMinstructionalimprovementprograms.WefocusedonSTEMbecauseofitsimportancetothenationaleconomy(Atkinson&Mayo,2010;NationalCommissiononMathematicsandScienceTeaching,2000),andbecausewewantedtoprovideevidenceonSTEM-specificprofessionallearningactivities,ratherthanmoregeneralactivities.BecausemanySTEMinstructionalimprovementprogramsweredevelopedin
![Page 5: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/5.jpg)
responsetocallstoincreasebothstudentunderstandingofcoredisciplinaryideasandstudentengagementwithkeydisciplinarypracticessuchasinquiry,argumentationandproof(NGA,2010;NCTM,2000;NRC2012),wereasonedthatsuchpracticeswouldnotnecessarilycarryacrossdisciplines.Asistypicalinareviewofthiskind,weconductedextensivedatabasesearches,combedthrougholderresearchsyntheses,andcontactedtheprincipalinvestigatorsofstudieswithunpublishedfindings.Werequiredstudiestoeitherfeaturerandomassignmentofteachersorschoolstothenewprogramoracontrolgroup,ortohaveamatchedcomparisongroupofteachersidentifiedbeforedatacollectionbegan(fordetails,seeLynch,Hill,Gonzalez&Pollard,2019).Intheend,welocated89studiesofprogramsthatcontainedprofessionaldevelopmentforteachers;ofthese,71alsocontainednewcurriculummaterialsforteacherstouseinclassrooms,suggestingthatprogramdevelopersoftenpairedprofessionaldevelopmentwithnewclassroommaterials.Becausemanyconsidercurriculummaterialstobeanimportantsourceofteacherprofessionallearninginandofthemselves(Ball&Cohen,1996),wealsoincludedsixstudiesofnewcurriculummaterialsbutnoassociatedprofessionaldevelopment;anotherthreestudiescomparedcurriculumwithandwithoutprofessionaldevelopmentandwereincludedinthecountabove.Welocated95studiesintotal.Wethenreadthroughthesestudiesandcreatedadatasetthatcontainedseveralpiecesofinformationfromeachstudy.
• Programtype:Focusonprofessionaldevelopment,curriculummaterials,orboth.• Assessmenttype:Studentoutcomesmeasuredviastateordistrictstandardizedtest,
otherstandardizedtest,orresearcher-designedassessment.Forteacherprofessionaldevelopment,wefurthercodedseveralotherfeatures:
• ThelengthofthePD;• Thefocus(orfoci),includingtopicslikeimprovingteachercontentknowledge,
integratingtechnologyintotheclassroom,andlearninghowtousecurriculummaterials;
• Theactivitiesthatteachersengagedinduringtheprofessionaldevelopment,suchasreviewingstudentwork,observingademonstrationofinstruction,orworkingthroughstudentcurriculummaterials;
• Theformatoftheprofessionaldevelopment,forinstancewhetheritwasdeliveredduringasummerworkshop,containedcoaching,orinvolvedonlinelearning.
Forcurriculummaterials,wecodedforthefractionoftheoriginalcurriculumitwastoreplace,whethertherewasanybackgroundinformationaboutcontentorstudentlearningembedded
![Page 6: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/6.jpg)
inthematerials,andwhetherthematerialscontainedkitsforstudentexperimentsandactivities.Followingcoding,wesetouttocalculateanaverageeffectsizeonstudenttestscoresacrossprograms,andtounderstandwhetheranyprogramcharacteristicspredictedstrongerorweakerstudentoutcomes.Weusedastatisticaltechniquecalledmeta-analysistodoso;wedescribethedetailsofthisanalysisinLynch,Hill,GonzalezandPollard(2019).ProfessionalLearningProgramsImproveStudentOutcomesFigure1showstheaverageeffectsizeacrossallSTEMinstructionalimprovementprograms,bothoverallandbrokendownbythetypeofassessmentusedintheevaluation.Acrossall95studies,theaverageprogramproducesa+8-percentiledifferenceintherankoftheaveragetreatment-andcontrol-groupstudent.Thiseffectismuchlarger(+14percentiles)forresearcher-designedassessmentsthanforstate-standardizedandotherstandardizedassessments(+2percentilesand+3percentiledifferences,respectively).Forallassessmenttypes,ouranalysisshowsthatthedifferencebetweenthetreatmentandcontrolgroupsisnotlikelytobezero–meaningtheseeffectsarestatisticallysignificant,thoughinthecaseofstandardizedassessments,notlarge.Acriticalquestionforouranalysiswaswhethersomeprogramfeaturesoutperformedothers,intermsofboostingstudentoutcomes.Figure2showsthatprogramspoststrongeraverageeffectswhentheyfeatureprofessionaldevelopmentpairedwithnewcurriculummaterials(anaverage+10percentilerankdifferencebetweentreatmentandcontrol)asopposedtofeaturingeithercurriculummaterialsorprofessionaldevelopmentalone(anaverage+6percentile-rankdifferencebetweentreatmentandcontrol).Figure3,whichshowstheaverageimpactsassociatedwithdifferentprofessionaldevelopmentprogramfoci,explainsthisfindinginmoredepth.Here,wecomparedprogramwithspecificcharacteristicstoseewhetheranyledtostrongerstudentoutcomesthanprogramswithoutthesecharacteristics.Wealsodescriptivelycomparedtheimpactsofprogramswiththesecharacteristicsrelativetotheaverageprogramimpactinthe95-studydataset.Twoprogramfoci–helpingteacherslearnhowtousecurriculummaterials,andimprovingteachers’contentknowledge,pedagogicalcontentknowledge,andknowledgeofstudentlearning–postedbetterstudentoutcomes(oftwoandthreepercentilepointsbetterthantheaverageprogram,respectively)thanprogramswithoutthesefoci.ThissuggeststhatthecombinedeffectofcurriculumandprofessionaldevelopmentinFigure2mayresultfromteacherslearninghowtousethematerialsandimprovingtheircontentknowledgeandknowledgeofstudentsalongtheway.Thishypothesisissupportedbyamorequalitativereadoftheincludedstudies;manythatfeaturedbothcurriculummaterialsandprofessionaldevelopmentoftenengagedteachersinsolvingmathematicsproblems,takingpartinscientificinvestigation,watchingfacilitatorsmodelinstruction,andstudyingstudentwork.Relationshipsbetweenotherfociandstudentoutcomeswerestillpositive,butouranalysisdidnotindicatetheyweredifferent,onaverage,thanprogramsthatdidnotincludethesefeatures.
![Page 7: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/7.jpg)
Inthecaseofintegratingtechnologyandcontent-specificformativeassessment,whichpostedstrongerabsolutegainsthanotherfoci,thisinabilitytodifferentiatetheeffectfromzerowasduetothesmallnumberofstudiesincludedinthesecategories.Figure4showsaverageimpactsassociatedwithdifferentprofessionaldevelopmentprogramformats.Threeformats–same-schoolcollaboration(+2percentiles),implementationmeetings(+4percentiles),andsummerworkshops(+2percentiles)–yieldedstrongergainsonstudentassessmentsthanprogramswithouttheseformats.Same-schoolcollaborationoccurredwhenteachersparticipatedintheprofessionaldevelopmentsessionalongsideotherteachersintheirschool,andimplementationmeetingsallowedteacherstoconvenebrieflywithotheractivityparticipantsafterthestartofimplementationtodiscussobstaclesandaidstoputtingtheprogramintopractice.Professionallearningwithanonlinecomponentyieldedlowerimpactsonstudentlearning(-4percentiles)thanprogramsthatwereentirelyfacetoface.Andprogramswithcoaching,apopularapproachtoinstructionimprovementinmanydistricts,yieldedimpactssimilartoprogramswithoutcoaching.However,fewprogramsfocusedonextended1:1coaching;instead,coachingappearedmoreasanadd-ontotraditionalprofessionaldevelopment.Ouranalysesdetectednopositiveornegativeassociationsbetweentheactivitiesteachersengagedduringprofessionaldevelopmentandthesizeofprogrameffects;thisincludedactivitiessuchasreviewingstudentwork,solvingproblems,developingcurriculummaterials,andreviewingbothgenericandtheirownstudents’work.Thesamewastrueforfeaturesofnewcurriculummaterials–nofeaturesofthosematerialsoutperformedothers.Finally,thedurationoftheprofessionaldevelopmentwasunrelatedtostudentoutcomes.Becausethiswascontrarytoourexpectation,soweconductedseveralanalysestoassesspossiblethresholdeffects(e.g.,morethan10hours)oracurvilinearrelationship(e.g.,professionaldevelopmentismaximallyeffectivewhenbetween20and40hours),butwefoundnoevidenceforeither.Afterreading95studiesdescribingSTEMinstructionalimprovementprograms,whatcanwesay?Thecharacteristicssignificantlyassociatedwithabove-averagestudentgainsincluded:
• Professionaldevelopmentfocusedonnewcurriculummaterials;• Programsaimedatimprovingteachers'content/pedagogicalcontentknowledge,
orunderstandingofhowstudentslearn;• Programscontainingspecificformats,including:
• Meetingstotroubleshootanddiscussclassroomimplementationoftheprogram;
• Summerworkshopsthatallowconcentratedlearningtime;• Same-schoolparticipationandcollaboration.
![Page 8: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/8.jpg)
Programswithonlysomeorfewofthesecharacteristicsmaystillhavepositiveeffects;however,whenprogramsdidincludethesecharacteristics,studentoutcomeswereimprovedabovetheaverageprogrameffect.Webelievethattheresultsofthismeta-analysishighlighttheimportanceofprofessionalknowledgeforteaching.Withotherscholars(Ball,Thames&Phelps2008;Shulman,1986;Lampert,2001),weviewprofessionalknowledgeinteachingasknowinghowthecontent,studentthinking,andcurriculumcometogether,andthenmakinggoodinstructionaldecisionsbasedontheparticularsofthesituation.Programsthatoutperformothersinouranalysistendedtofocusongrowingthisformofknowledgeasopposedtogeneralpedagogicalknowledge,ormoreperipheraltopicsliketechnology.Whatcan’twesay?Meta-analyseshavetheadvantageofexaminingprogramsimplementedacrossawidevarietyofcontexts,providingsomerobustnesstofindings.However,thismeta-analysisislimitedinwhatitcansayaboutprofessionallearningsystems“ontheground”inU.S.schoolsanddistricts.Tostart,eachprogramweexaminedwasimplementedinaspecificcontext,orasmallsetofcontexts;whethertheprogramwouldsucceedinanothercontextisanopenquestion.Critically,ananalysisfoundaslighttrendtowardsmallerimpactsoftheseprogramsinhigh-povertysettings,suggestingthatinterventionsmayworkbetter,onaverage,indistrictsservingmoreadvantagedstudents.However,wefoundnofurtherinteractionsbystudentrace,ethnicity,districttype(urban,suburbanorrural),andsizeofthetreatmentgroup.Thatsaid,otheraspectsofdistrictandschoolcontextmayinteractwithprogramefficacy.Weknowfromstudiesofpolicyimplementation,forinstance,thatleadershipandpeersupportmatterquiteabitinencouragingteachertake-upanduseofnewinstructionalpractices(e.g.,Matsumura,Garnier,&Resnick,2010;Wanless,Patton,Rimm-Kaufman&Deutsch2013),andthepresenceofcompetinginstructionalguidanceandinitiatives(e.g.,instructionalpacingguides,conflictingadviceonwhatandhowtoteach)tendstodampenteacherchange(Hill,Corey&Jacob,2018).Thestudieswereviewedcontainednoinformationaboutthesefactors,however,sowecouldnottestthemformallyinourmodels.Second,theprogramsdescribedheretendedtobesmall,intensive,enrolledvolunteerteachers,andwereoftenledbyuniversityacademicsorresearchers.Bycontrast,localprofessionaldevelopmentcancontainmyriaddifferentofferings,withteachersspreadingtheirtimeacrossseveraldifferentsettings(summerworkshops,grade-levelteammeetings),topics(ELA,mathematics),andinsessionsledbyotherteachersorschoolordistrictleaders.Insomesystems,teachershaveatleastpartialchoiceovertheprofessionaldevelopmenttheyengage,whileinothersystemstheyhaveverylittle.Oneimplicationofthesedifferencesbetweenthestudysampleandtypicalpracticeisthatwedon’tyetknowwhetherthefeaturesthat“work”inouranalysiswillworkintypicalU.S.schools.Doesaprogramfocusoncontentandpedagogicalknowledgeimprovestudentoutcomes?Doesworkingthroughnewcurriculummaterialsyieldbenefitstotheaverage
![Page 9: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/9.jpg)
teacher?Alotdependsuponthequalityoflocalimplementation,andinourestimation,howdeeplyteachersengagewiththesubjectmatterandattempttoimprovetheircraft.STEM-focusedprofessionallearninginwidercontextAsnotedintheintroduction,teachersofSTEMengageinawidevarietyofprofessionallearningactivities,ofteninasingleyear.Thisleadstoanimportantquestionfordistricts:howtomakemoretimeforthekindsoflearningopportunitiesthatpostedbettergains?Teachersalreadyreportfeelingoverwhelmedbythesheervolumeofreformandever-increasinginstructionalresponsibilities(AFT,2017;Valli&Beuse,2007),andit’slikelythatscalingbackoreliminatinganactivitywillbenecessarytomakeroomformoreefficaciousformsofprofessionallearning.Basedonevidenceintheliterature,wewouldnominateeliminating“datateammeetings,”1whereteachersstudystudentdatainhopesofindividualizingandimprovinginstruction.Inareviewseparatefromthisone,nineevaluationsofprogramsthatfeaturedteachers’studyofdataproducedonlytwopositive(andonenegative)resultslikelytobestatisticallydifferentfromzero,outofatotal19impactanalysesrelatingprogramparticipationtostudenttestscoreoutcomes.Aswell,qualitativeresearchsuggestthatteachersstudyingdatadoesnotitselfleadtoneworimprovedinstructionaltechniques(Barmore,2018;Goertz,Oláh,L.N.,&Riggan,2009),andourownobservationsofandparticipationinschool-baseddatateamssuggeststhatteamdiscussionsoftenascribepoorstudentperformancetofactorsotherthaninstructionitself(e.g.,alackofbackgroundknowledge,abadweek,troubleathome)(seealsoGoertzetal.,2009).Yetrecentnationalsurveysoflocalprofessionaldevelopmentsuggestthatschoolshavemadelargeinvestmentsinhavingteachersstudystudentassessmentdata(e.g.,Banilower,etal.,2018).Repurposingthesemeetingstowardbuildingexpertiseincurriculummaterialsandcontentseemsnatural;wecaution,however,thatdistrictswillhavetodosocarefully,usingroutinesandstructuresthatfocusattentionsquarelyandindepthoninstruction.ConclusionThattheseSTEMinstructionalimprovementprogramsbooststudentoutcomesshouldbeareasonforoptimismamongpolicymakersandleaders.Ourfindingsmay,forinstance,helpshapehowstatesanddistrictschoosetospendTitleIIdollars,fundsaimedatimprovingteacherquality.Theyalsosuggesthowleadersmaynarrowthescopeofteacherprofessionallearninginwayslikelytoincreasetheefficiencyandimpactofthoseefforts.Finally,ourfindingsalsosuggesttheimportanceofteachers’professionalknowledgetostudentoutcomes,ahypothesessupportedbystudieslinkingteacherknowledgetostudentoutcomes(e.g.,Baumert,2010;Hill,Rowan&Ball,2005)andthathasimplicationsforteacherhiringandretention.
1Wedifferentiate“datause”programsfrom“formativeassessment”programsinthattheformertypicallyusesdatafrominterimorbenchmarktestswhilethelatterhelpsteacherscreatetheirownassessments,eitherfromitembanks,curricularassessments,orbasedonprinciplesofgoodassessment.Programsthatfeaturedcontent-specificformativeassessmentwereincludedinouranalysis.
![Page 10: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/10.jpg)
ReferencesAmericanFederalofTeachers(2017).2017educatorqualityoflifesurvey.Washington,DC:AFT.Atkinson,RobertD.andMayo,MerrileaJoyce,RefuelingtheU.S.InnovationEconomy:FreshApproachestoScience,Technology,EngineeringandMathematics(STEM)Education(December9,2010).TheInformationTechnology&InnovationFoundation,Forthcoming.AvailableatSSRN:https://ssrn.com/abstract=1722822Ball,D.L.,&Cohen,D.K.(1996).Reformbythebook:Whatis—ormightbe—theroleofcurriculummaterialsinteacherlearningandinstructionalreform?EducationalResearcher,25(9),6-14.Ball,D.L.,Thames,M.H.,&Phelps,G.(2008).Contentknowledgeforteaching:Whatmakesitspecial.JournalofTeacherEducation,59(5),389-407.Banilower,E.R.,Smith,P.S.,Malzahn,K.A.,Plumley,C.L.,Gordon,E.M.,&Hayes,M.L.(2018).Reportofthe2018NSSME+.ChapelHill,NC:HorizonResearch,Inc.Barmore,J.M.(2018).JourneyfromDataintoInstruction:HowTeacherTeamsEngageinData-DrivenInquiry.Cambridge,MA:HarvardUniversity.Baumert,J.,Kunter,M.,Blum,W.,Brunner,M.,Voss,T.,Jordan,A.,...&Tsai,Y.M.(2010).Teachers’mathematicalknowledge,cognitiveactivationintheclassroom,andstudentprogress.AmericanEducationalResearchJournal,47(1),133-180.Gersten,R.,Taylor,M.J.,Keys,T.D.,Rolfhus,E.,&Newman-Gonchar,R.(2014).Summaryofresearchontheeffectivenessofmathprofessionaldevelopmentapproaches.(REL2014–010).Washington,DC:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance,RegionalEducationalLaboratorySoutheast.Retrievedfromhttp://ies.ed.gov/ncee/edlabsGoertz,M.E.,Oláh,L.N.,&Riggan,M.(2009).Fromtestingtoteaching:Theuseofinterimassessmentsinclassroominstruction.Philadelphia,PA:UniversityofPennsylvania.Hill,H.C.,Corey,D.L.,&Jacob,R.T.(2018).DividingbyZero:ExploringNullResultsinaMathematicsProfessionalDevelopmentProgram.TeachersCollegeRecord,120(6),n6.Hill,H.C.,Rowan,B.,&Ball,D.L.(2005).Effectsofteachers’mathematicalknowledgeforteachingonstudentachievement.AmericanEducationalResearchJournal,42(2),371-406.Lampert,M.(2001).Teachingproblemsandtheproblemsofteaching.YaleUniversityPress.
![Page 11: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/11.jpg)
Lynch,K.,Hill,H.C.,Gonzalez,K.E.,&Pollard,C.(2019).StrengtheningtheResearchBasethatInformsSTEMInstructionalImprovementEfforts:AMeta-Analysis.EducationalEvaluationandPolicyAnalysis,0162373719849044.Matsumura,L.C.,Garnier,H.E.,&Resnick,L.B.(2010).Implementingliteracycoaching:Theroleofschoolsocialresources.EducationalEvaluationandPolicyAnalysis,32(2),249-272.NationalCommissiononMathematicsandScienceTeachingforthe21stCentury(US),&ChairGlenn.(2000).Beforeit'stoolate:AreporttothenationfromtheNationalCommissiononMathematicsandScienceTeachingforthe21stCentury.UnitedStates.U.S.DepartmentofEducation,Washington,D.C.Retrievedfromhttps://files.eric.ed.gov/fulltext/ED441705.pdfNationalCouncilofTeachersofMathematics.(2000)PrinciplesandStandardsforSchoolMathematics.Reston,VA:NCTMNationalGovernorsAssociation,CouncilofChiefStateSchoolOfficers.(2010).CommonCoreStateStandards.Washington,DC:Author.NationalResearchCouncil,BoardonScienceEducation.(2012).AframeworkforK-12scienceeducation:Practices,crosscuttingconcepts,andcoreideas.Washington,DC:TheNationalAcademiesPress.Retrievedfromhttp://www.nap.edu/catalog.php?record_id=13165Riener,C.,&Willingham,D.(2010).Themythoflearningstyles.Change:Themagazineofhigherlearning,42(5),32-35.Shavelson,R.J.,&Towne,L.(Eds.).(2001).Scientificresearchineducation.Washington,DC:TheNationalAcademiesPress.Wanless,S.B.,Patton,C.L.,Rimm-Kaufman,S.E.,&Deutsch,N.L.(2013).Setting-levelinfluencesonimplementationoftheResponsiveClassroomapproach.PreventionScience,14(1),40-51.Yoon,K.S.,Duncan,T.,Lee,S.W.-Y.,Scarloss,B.,&Shapley,K.(2007).Reviewingtheevidenceonhowteacherprofessionaldevelopmentaffectsstudentachievement(Issues&AnswersReport,REL2007–No.033).Washington,DC:U.S.DepartmentofEducation,InstituteofEducationSciences,NationalCenterforEducationEvaluationandRegionalAssistance,RegionalEducationalLaboratorySouthwest.Retrievedfromhttp://ies.ed.gov/ncee/edlabs
![Page 12: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/12.jpg)
Figure1
![Page 13: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/13.jpg)
Figure2
![Page 14: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/14.jpg)
Figure3
![Page 15: STEM Instruction Improvement Programs Improve Student … · response to calls to increase both student understanding of core disciplinary ideas and student engagement with key disciplinary](https://reader034.fdocuments.us/reader034/viewer/2022052003/6016497f0c3b5c19986caeaa/html5/thumbnails/15.jpg)
Figure4