Philip C. Abrami, Anne Wade, Larysa Lysenko Jonathon Marsh ......Elementary School Students: A Brief...
9
The Effects of ABRACADABRA Early Literacy Software on the Learning of Kenyan Elementary School Students: A Brief Report on the 2015 Study 123 Philip C. Abrami, Anne Wade, Larysa Lysenko Centre for the Study of Learning & Performance Concordia University, Montreal, Canada Jonathon Marsh, Aga Khan Academies Unit, Paris, France & Gioko Maina, Aga Khan Academy, Mombasa REVISED: February 17, 2017 Background In 2012, the Centre for the Study of Learning and Performance (CSLP) joined with the Aga Khan Academies unit and the Aga Khan Academy, Mombasa (AKA) and they together embarked on a preliminary investigation to explore the feasibility and impact of using ABRACADABRA (ABRA), early literacy software on grade-one students in Mombasa, Kenya. Children from local schools were bussed to the Aga Khan Academy, Mombasa and used the Academy computer lab to access ABRA. For a period of several months about once per week for up to two hours they used the software under the careful supervision of trained teachers and staff. Control students received traditional instruction following the national curriculum. At the conclusion of the program, ABRA students showed more significant gains on GRADE, a standardized test of reading skills, than their peers in control classes. ABRA students also scored higher on several Kenyan core subject examination including English, Math, Science and Social Studies. In 2013, the control students used ABRA and they also showed improvements in literacy skills. In 2014, the research was extended but with an important difference. Instead of bringing the children to the Academy labs to use ABRA under the supervision of staff, the project took place in school computer labs. The experimental teachers were trained in the use ABRA and learned strategies on how to integrate the software into their English Language instruction. They were also provided with ongoing support by means of bi-weekly planning sessions and regular school visits. ABRA ambassadors, seasoned master teachers who had 1 This research was made possible by a grant to the authors from the SESEA project, Aga Khan Foundation Canada and the SSHRC Partnership Development program. 2 The authors express their appreciation to the Kenyan coordinators and ambassadors who worked extensively to train and support the teachers who implemented ABRACADABRA in their classrooms. The authors are also grateful to the CAMARA organization who outfitted the school computer labs and provided regular maintenance during the ABRA intervention. 3 For additional information about ABRA or any of the software tools in the Learning Toolkit please contact Anne Wade at [email protected].
Transcript of Philip C. Abrami, Anne Wade, Larysa Lysenko Jonathon Marsh ......Elementary School Students: A Brief...
TheEffectsofABRACADABRAEarlyLiteracySoftwareontheLearningofKenyan
ElementarySchoolStudents:ABriefReportonthe2015Study123
PhilipC.Abrami,AnneWade,LarysaLysenkoCentrefortheStudyofLearning&Performance
ConcordiaUniversity,Montreal,Canada
JonathonMarsh,AgaKhanAcademiesUnit,Paris,France&GiokoMaina,AgaKhanAcademy,Mombasa
REVISED:February17,2017Background
In2012,theCentrefortheStudyofLearningandPerformance(CSLP)joinedwiththeAgaKhanAcademiesunitandtheAgaKhanAcademy,Mombasa(AKA)andtheytogetherembarkedonapreliminaryinvestigationtoexplorethefeasibilityandimpactofusingABRACADABRA(ABRA),earlyliteracysoftwareongrade-onestudentsinMombasa,Kenya.ChildrenfromlocalschoolswerebussedtotheAgaKhanAcademy,MombasaandusedtheAcademycomputerlabtoaccessABRA.Foraperiodofseveralmonthsaboutonceperweekforuptotwohourstheyusedthesoftwareunderthecarefulsupervisionoftrainedteachersandstaff.Controlstudentsreceivedtraditionalinstructionfollowingthenationalcurriculum.Attheconclusionoftheprogram,ABRAstudentsshowedmoresignificantgainsonGRADE,astandardizedtestofreadingskills,thantheirpeersincontrolclasses.ABRAstudentsalsoscoredhigheronseveralKenyancoresubjectexaminationincludingEnglish,Math,ScienceandSocialStudies.In2013,thecontrolstudentsusedABRAandtheyalsoshowedimprovementsinliteracyskills.
In2014,theresearchwasextendedbutwithanimportantdifference.InsteadofbringingthechildrentotheAcademylabstouseABRAunderthesupervisionofstaff,theprojecttookplaceinschoolcomputerlabs.TheexperimentalteachersweretrainedintheuseABRAandlearnedstrategiesonhowtointegratethesoftwareintotheirEnglishLanguageinstruction.Theywerealsoprovidedwithongoingsupportbymeansofbi-weeklyplanningsessionsandregularschoolvisits.ABRAambassadors,seasonedmasterteacherswhohad
1ThisresearchwasmadepossiblebyagranttotheauthorsfromtheSESEAproject,AgaKhanFoundationCanadaandtheSSHRCPartnershipDevelopmentprogram.2TheauthorsexpresstheirappreciationtotheKenyancoordinatorsandambassadorswhoworkedextensivelytotrainandsupporttheteacherswhoimplementedABRACADABRAintheirclassrooms.TheauthorsarealsogratefultotheCAMARAorganizationwhooutfittedtheschoolcomputerlabsandprovidedregularmaintenanceduringtheABRAintervention.3ForadditionalinformationaboutABRAoranyofthesoftwaretoolsintheLearningToolkitpleasecontactAnneWadeatwada@education.concordia.ca.
ABRA2015Study
2
startedusingABRAasearlyas2012,facilitatedthesesupportactivities.ThistimetheexperimentalparticipantsusedtheABRAsoftwareoverseveralmonths,withthecontrolparticipantsnotusingABRA.TheGRADEreadingtestthatthestudentstookwhentheinterventionwascompletedalsoshowedthattheABRAstudentsoutperformedtheirpeerswhodidnotuseABRA.The2015Experiment
The2015study,whichwearebrieflyreportinghere,wasalsoconductedinpublicschoolsinMombasaandNairobi.Similarto2014,teachersimplementedtheABRAinterventionintheirclassroomsforseveralmonths.Butkeydifferencesincludedtestingallstudentsatthebaseline,beforetheyusedABRAaswellastestingthemattheconclusionoftheintervention.Inaddition,wewereabletoestablishpretestequivalenceoftheABRAstudentsversusthecontrolstudents.Inotherwords,the2015studywasastrongertestofABRAconductedundertheauthenticconditionsofKenyanclassroomuse.Thisstudywasapretest-posttestcontrolgroupquasi-experimentaldesignwherewestatisticallycontrolledfortheinitialdifferencesbetweenABRAandcontrolstudents.Therewereatotalof22teachers(11ABRAexperimentalteachers,11controlteachers)andtheir749students(307ABRAstudents,442controlstudents)Studentswerefromgrades1,2&3.SomehadusedABRAthepreviousyear,whileotherswerefirsttimeusers.Wealsorecordedstudentgendersothatwecouldcomparechangesinliteracyscoresfor366boysversus383girls.InadditiontocollectingGRADEstandardreadingtotalscoreandsubtestscores,wealsocollectedandanalyzedKenyannationalexaminationresults.Finally,weusedanumberofdifferentstatisticalstrategiestoanalyzethescores.Theresultswereconsistentacrossalltheseanalyses.ResultsFirst,wecomparedABRA(N=307)andcontrol(N=442)groupsatthepretest.ThistestofgroupdifferencesdidnotrevealanysignificantdifferencesbetweentheABRAandcontrolgroupsontheGRADEsubtestsandtotaltestscores(pvocabulary>.09;pcomprehension>.26;ptotaltest>.7;plisteningcomprehension>.06).Overall,theseresultslendcredibilitytothenotionthattheexperimentalandcontrolstudentsweremostlyequivalentinliteracyskillsattheoutsetoftheABRAintervention.AsummaryoftheANOVAresultsofthedifferencescoresonallthesubscalesoftheGRADEtestsisreportedinTable1.Table1.GRADEmeans,standarddeviations,changescoresanddifferencebetweenABRAandcontrolstudents
GRADEscalesABRAmeans(N=307) Controlmeans(N=442)
Fvalueandsignificance(***p<0.00)Post Pre Change Post Pre Change
VocabularyComposite 44.42 41.77 2.65 40.86 42.98 -2.11 53.08***
StandardDeviation 9.88 10.86 7.50 9.46 8.88 9.61
ReadingComprehension 22.56 17.48 5.08 17.21 16.83 0.38 62.71***
StandardDeviation 9.25 7.75 8.94 8.01 7.67 7.28
ABRA2015Study
3
TheresultsshowthatonGRADEsubtestsandTotalTest,studentsinABRAclassesimprovedtheirscoresatasignificantlyhigherratethanstudentsincontrolclasses.Theconsiderableeffectsizeof0.58fortheGRADETotalTestindicatedthatafterhavingusedABRA,averagestudentsintheABRAgroupimprovedby22percentilepointsovertheirpeersinthecontrolgroupwhowereexposedtotraditionalinstruction.Figure1belowrepresentsthedifferenceinmeanchangescoresbetweenthetwogroupsontheGRADETotalscoreovertime.
Figure1.GRADEtestTotalReadingAverageScores(changeovertime)
GenderInadditiontothemainanalysis,weexaminedifABRAeffectsdifferedforboysandgirls.Table2andFigure2splittheGRADETotalTestscoresforboysandgirlsintheABRAandcontrolclasses.ThedatasuggestthatgirlsandboysbenefitedfromABRAwithABRAgirlsdemonstratinghighergainsthanABRAboys.BothgendersintheABRAclassesshowedenhancedperformanceontheGRADEcomparedtostudentslearningtoreadinthetraditionalmanner,withafocusonteacher-directedandrecitationinstruction.
2015 Change in GRADE(Total Test) scores: experimental vs. control
50
55
60
65
70
75
Pretest(Winter,2015) Pos4est(Fall,2015)
Control(N=442)
ABRA(N=307)
TotalTest 66.98 59.54 7.70 58.07 59.73 -1.65 81.58***StandardDeviation 17.74 15.58 13.35 15.92 14.83 14.33 ListeningComprehension 10.91 8.5 2.40 9.48 9.09 0.38 36.60***StandardDeviation 2.89 2.78 3.41 2.98 4.85 5.13
ABRA2015Study
4
Table2.GRADETotalTestMeanScoresbyGender
Figure2.ABRAEffectsSeparatedbyGender
KenyaExamResultsToexamineiftheABRAeffectswereassociatedwithstudents’learninginthecorecurriculumsubjects,weusedtheresultsofend-of-termexamsprovidedbyKenyateachers.Therewasconsiderabledropinthesamplesizebecauseexamresultsofonly464studentshavebeenfurnishedforanalyses.WecomparedthescoresinEnglish,Math,ScienceandSocialStudiesofcontrolandABRAstudentscollectedattheendoftermthreeafterstatisticallyadjustingthesescoresusingtheGRADEpre-testtotalscoreasthecovariate.TheanalyseswereperformedforEnglish,Math,ScienceandSocialStudiesexamscores.TheassociationbetweenGRADEpretestscoresandtheexamscoreswasstatisticallysignificantandrangedfrommodestof.28(Math)to.6(English).
2015 Change in GRADE(Total Test) scores: Gender
50
55
60
65
70
75
Pretest(Winter,2015) Pos4est(Fall,2015)
Controlboys(N=230)
Controlgirls(N=212)
ABRAboys(N=136)
ABRAgirls(N=171)
GRADEscores
ABRA Control
Girls(N=171) Boys(N=136) Girls(N=212) Boys(N=230)
Post Pre Post Pre Post Pre Post Pre
TotalTest 70.53 60.13 62.53 58.21 58.85 60.6 57.36 58.92
StandardDeviation 16.20 16.48 18.63 16.79 16.13 15.04 15.73 14.12
ABRA2015Study
5
ThetestsfortheeffectofABRAonEnglish,Math,ScienceandSocialStudiesexamscoresshowedthedifferencebetweenthegroupswasstatisticallysignificant(p<0.001).ThestatisticssummarizedinTable3andillustratedinFigure3showthatstudentsintheABRAgroupperformedhigherthancontrolstudentsoneachofthefourcoreexams.Table3.CoreexamsresultsforABRAandcontrolclasses(adjustedmeans,andstandarderrorvalues)
1Adjustedmeansforpost-testscoresandstandarderrorcalculatedinthemodel.GRADEpre-testcovariatescoresmodelwereevaluatedat58.18.
Coresubjectexams ABRAMeans1(N=149)
ControlMeans1(N=315)
English 85.86 74.03StandardError 1.34 0.91Mathematics 77.40 68.31StandardError 1.58 1.08Science 86.39 68.87StandardError 1.23 0.84SocialStudies 82.25 65.14StandardError 1.20 0.83
ABRA2015Study
6
Figure3.KenyaEndofYearExaminationResultsbySubjectGradeLevelStudentgradeleveldidnotlessentheeffectsofABRA.StudentsinGrades1,2&3allbenefitedfromABRAregardlessofwhethertheyhadusedABRAinthepreviousschoolyear.Table4andFigure4allowonetofollowhowthemeanGRADETotalTestscoreschangedfrompre-topost-testbygradeinABRAandcontrolgroups.AsexpectedthehighestincreaseisnoticeableforABRAstudentsingrades2and3asrespectively23%and85%ofstudentsinthesegradeshadbeenexposedtoABRAsince2014for1ormoreyears.Table4.GRADETotalTestMeanScoresbyGradeLevel
Difference between the groups: Core end-of-year 2015 exams
10
20
30
40
50
60
70
80
90
English Socialscience Mathema;cs Science
Control(N=315)
ABRA(N=149)
GRADEscores
ABRA ControlGrade1(N=83)
Grade2(N=124)
Grade3(N=100)
Grade1(N=98)
Grade2(N=151)
Grade3(N=193)
Post Pre Post Pre Post Pre Post Pre Post Pre Post Pre
TotalTest 47.58 44.01 69.85 59.6 79.54 71.56 47.37 50.93 51.79 57.46 68.43 65.96
StandardDeviation 14.38 14.95 12.12 13.33 11.57 9.93 8.97 12.26 13.05 14.63 14.53 13.39
ABRA2015Study
7
Figure4.ResultsbyStudentGradeLevelTeacherExperienceWealsoexploredwhetherthereweredifferencesamongteachersbasedonyearsofexperiencesusingABRA.Becauseofinitialtrainingandfollow-upsupportaswellasin-classexperienceoveroneormoreyears,wedidnotexpectthatnoviceuserswouldhavelowereffectscomparedtoexperiencedusers.However,wewereconcernedthatiftherewerenoveltyeffectstheywoulddissipateovertime.However,asFigure5shows,therewerenodecrementsinstudentlearninglinkedtoteacherexperience.
Difference between the groups on GRADE(Total Test) post-test scores: grade level
20
30
40
50
60
70
grade1(Ncontrol=98;NAbra=83)
grade2(Ncontrol=151;NAbra=124)
grade3(Ncontrol=193;NAbra=100)
Control
ABRA
ABRA2015Study
8
Figure5.YearsofTeacherExperienceandABRAEffectsImplications
Theseremarkableresultsareatestamenttotheevidence-baseddesignofABRAandqualityteacherimplementationintheauthenticcontextoftheirschools.ABRAisnotintendedtosupplementclassroominstruction,butinsteadshouldbeintegratedintoclassroominstruction.ThetrainingandsupportprovidedbytheABRAcoordinatorsandambassadorscoupledwithteacherswillingtoworktoimproveteachingarewhatmadethedifference.
SeveralotherimprovementsshouldmaketheefficacyofABRAevenstrongerinyearstocome.First,weexpandedandimprovedourteacherprofessionaldevelopmentresourcesandassembledthemtogetherinTheLearningToolkitTeachersGuide:KenyaEdition.Second,toenhancefurtherstudentfluencyandcomprehensionskills,wedevelopedREADS,afreelyavailabledigitalrepositoryofhundredsofbooksandstoriesinmultiplelanguagesincludingEnglishandKiswahili.READSfitseasilyonaUSBkeyandmeansnoKenyanchildwilllackengagingreadingmaterialtopracticeandhonetheirliteracyskills.Third,wemorecompletelymappedABRAactivitiesandstoriesontotheKenyancurriculumsothatteacherswillfindintegratingABRAintotheirlessonsespeciallyeasy.Fourth,wetackledthechallengeoflargeclasssizesandtherelianceonfrontalteachingbyemphasizingcooperativesmallgrouptechniquesduringABRAlessons.Finally,to
Change in GRADE(Total Test) scores: teacher experience with ABRA
50
55
60
65
70
75
Pretest(Winter,2015) Pretest(Fall,2015)
Oneyear(N=68,grades1,and3)
Twoyears(N=157,grades1,2,and3)
Threeyearsandmore(N=68,grades2and3)
ABRA2015Study
9
complementgender-balancedstoriesandactivitiesinABRA,weofferedsupportmaterialstoteacherstohelpthemadequatelycultivategenderequalityintheirclassrooms.
WeareexcitedaboutthelatestABRAresearchandhopeyouaretoo.Wethinkwe’vetakenanothersteptocreatinganationofreadersinKenyaandbeyond.
