TheEffectsofABRACADABRAEarlyLiteracySoftwareontheLearningofKenyan

ElementarySchoolStudents:ABriefReportonthe2015Study123

PhilipC.Abrami,AnneWade,LarysaLysenkoCentrefortheStudyofLearning&Performance

ConcordiaUniversity,Montreal,Canada

JonathonMarsh,AgaKhanAcademiesUnit,Paris,France&GiokoMaina,AgaKhanAcademy,Mombasa

REVISED:February17,2017Background

In2012,theCentrefortheStudyofLearningandPerformance(CSLP)joinedwiththeAgaKhanAcademiesunitandtheAgaKhanAcademy,Mombasa(AKA)andtheytogetherembarkedonapreliminaryinvestigationtoexplorethefeasibilityandimpactofusingABRACADABRA(ABRA),earlyliteracysoftwareongrade-onestudentsinMombasa,Kenya.ChildrenfromlocalschoolswerebussedtotheAgaKhanAcademy,MombasaandusedtheAcademycomputerlabtoaccessABRA.Foraperiodofseveralmonthsaboutonceperweekforuptotwohourstheyusedthesoftwareunderthecarefulsupervisionoftrainedteachersandstaff.Controlstudentsreceivedtraditionalinstructionfollowingthenationalcurriculum.Attheconclusionoftheprogram,ABRAstudentsshowedmoresignificantgainsonGRADE,astandardizedtestofreadingskills,thantheirpeersincontrolclasses.ABRAstudentsalsoscoredhigheronseveralKenyancoresubjectexaminationincludingEnglish,Math,ScienceandSocialStudies.In2013,thecontrolstudentsusedABRAandtheyalsoshowedimprovementsinliteracyskills.

In2014,theresearchwasextendedbutwithanimportantdifference.InsteadofbringingthechildrentotheAcademylabstouseABRAunderthesupervisionofstaff,theprojecttookplaceinschoolcomputerlabs.TheexperimentalteachersweretrainedintheuseABRAandlearnedstrategiesonhowtointegratethesoftwareintotheirEnglishLanguageinstruction.Theywerealsoprovidedwithongoingsupportbymeansofbi-weeklyplanningsessionsandregularschoolvisits.ABRAambassadors,seasonedmasterteacherswhohad

1ThisresearchwasmadepossiblebyagranttotheauthorsfromtheSESEAproject,AgaKhanFoundationCanadaandtheSSHRCPartnershipDevelopmentprogram.2TheauthorsexpresstheirappreciationtotheKenyancoordinatorsandambassadorswhoworkedextensivelytotrainandsupporttheteacherswhoimplementedABRACADABRAintheirclassrooms.TheauthorsarealsogratefultotheCAMARAorganizationwhooutfittedtheschoolcomputerlabsandprovidedregularmaintenanceduringtheABRAintervention.3ForadditionalinformationaboutABRAoranyofthesoftwaretoolsintheLearningToolkitpleasecontactAnneWadeatwada@education.concordia.ca.

ABRA2015Study

2

startedusingABRAasearlyas2012,facilitatedthesesupportactivities.ThistimetheexperimentalparticipantsusedtheABRAsoftwareoverseveralmonths,withthecontrolparticipantsnotusingABRA.TheGRADEreadingtestthatthestudentstookwhentheinterventionwascompletedalsoshowedthattheABRAstudentsoutperformedtheirpeerswhodidnotuseABRA.The2015Experiment

The2015study,whichwearebrieflyreportinghere,wasalsoconductedinpublicschoolsinMombasaandNairobi.Similarto2014,teachersimplementedtheABRAinterventionintheirclassroomsforseveralmonths.Butkeydifferencesincludedtestingallstudentsatthebaseline,beforetheyusedABRAaswellastestingthemattheconclusionoftheintervention.Inaddition,wewereabletoestablishpretestequivalenceoftheABRAstudentsversusthecontrolstudents.Inotherwords,the2015studywasastrongertestofABRAconductedundertheauthenticconditionsofKenyanclassroomuse.Thisstudywasapretest-posttestcontrolgroupquasi-experimentaldesignwherewestatisticallycontrolledfortheinitialdifferencesbetweenABRAandcontrolstudents.Therewereatotalof22teachers(11ABRAexperimentalteachers,11controlteachers)andtheir749students(307ABRAstudents,442controlstudents)Studentswerefromgrades1,2&3.SomehadusedABRAthepreviousyear,whileotherswerefirsttimeusers.Wealsorecordedstudentgendersothatwecouldcomparechangesinliteracyscoresfor366boysversus383girls.InadditiontocollectingGRADEstandardreadingtotalscoreandsubtestscores,wealsocollectedandanalyzedKenyannationalexaminationresults.Finally,weusedanumberofdifferentstatisticalstrategiestoanalyzethescores.Theresultswereconsistentacrossalltheseanalyses.ResultsFirst,wecomparedABRA(N=307)andcontrol(N=442)groupsatthepretest.ThistestofgroupdifferencesdidnotrevealanysignificantdifferencesbetweentheABRAandcontrolgroupsontheGRADEsubtestsandtotaltestscores(pvocabulary>.09;pcomprehension>.26;ptotaltest>.7;plisteningcomprehension>.06).Overall,theseresultslendcredibilitytothenotionthattheexperimentalandcontrolstudentsweremostlyequivalentinliteracyskillsattheoutsetoftheABRAintervention.AsummaryoftheANOVAresultsofthedifferencescoresonallthesubscalesoftheGRADEtestsisreportedinTable1.Table1.GRADEmeans,standarddeviations,changescoresanddifferencebetweenABRAandcontrolstudents

GRADEscalesABRAmeans(N=307) Controlmeans(N=442)

Fvalueandsignificance(***p<0.00)Post Pre Change Post Pre Change

VocabularyComposite 44.42 41.77 2.65 40.86 42.98 -2.11 53.08***

StandardDeviation 9.88 10.86 7.50 9.46 8.88 9.61

ReadingComprehension 22.56 17.48 5.08 17.21 16.83 0.38 62.71***

StandardDeviation 9.25 7.75 8.94 8.01 7.67 7.28

ABRA2015Study

3

TheresultsshowthatonGRADEsubtestsandTotalTest,studentsinABRAclassesimprovedtheirscoresatasignificantlyhigherratethanstudentsincontrolclasses.Theconsiderableeffectsizeof0.58fortheGRADETotalTestindicatedthatafterhavingusedABRA,averagestudentsintheABRAgroupimprovedby22percentilepointsovertheirpeersinthecontrolgroupwhowereexposedtotraditionalinstruction.Figure1belowrepresentsthedifferenceinmeanchangescoresbetweenthetwogroupsontheGRADETotalscoreovertime.

Figure1.GRADEtestTotalReadingAverageScores(changeovertime)

GenderInadditiontothemainanalysis,weexaminedifABRAeffectsdifferedforboysandgirls.Table2andFigure2splittheGRADETotalTestscoresforboysandgirlsintheABRAandcontrolclasses.ThedatasuggestthatgirlsandboysbenefitedfromABRAwithABRAgirlsdemonstratinghighergainsthanABRAboys.BothgendersintheABRAclassesshowedenhancedperformanceontheGRADEcomparedtostudentslearningtoreadinthetraditionalmanner,withafocusonteacher-directedandrecitationinstruction.

2015 Change in GRADE(Total Test) scores: experimental vs. control

50

55

60

65

70

75

Pretest(Winter,2015) Pos4est(Fall,2015)

Control(N=442)

ABRA(N=307)

TotalTest 66.98 59.54 7.70 58.07 59.73 -1.65 81.58***StandardDeviation 17.74 15.58 13.35 15.92 14.83 14.33 ListeningComprehension 10.91 8.5 2.40 9.48 9.09 0.38 36.60***StandardDeviation 2.89 2.78 3.41 2.98 4.85 5.13

ABRA2015Study

4

Table2.GRADETotalTestMeanScoresbyGender

Figure2.ABRAEffectsSeparatedbyGender

KenyaExamResultsToexamineiftheABRAeffectswereassociatedwithstudents’learninginthecorecurriculumsubjects,weusedtheresultsofend-of-termexamsprovidedbyKenyateachers.Therewasconsiderabledropinthesamplesizebecauseexamresultsofonly464studentshavebeenfurnishedforanalyses.WecomparedthescoresinEnglish,Math,ScienceandSocialStudiesofcontrolandABRAstudentscollectedattheendoftermthreeafterstatisticallyadjustingthesescoresusingtheGRADEpre-testtotalscoreasthecovariate.TheanalyseswereperformedforEnglish,Math,ScienceandSocialStudiesexamscores.TheassociationbetweenGRADEpretestscoresandtheexamscoreswasstatisticallysignificantandrangedfrommodestof.28(Math)to.6(English).

2015 Change in GRADE(Total Test) scores: Gender

50

55

60

65

70

75

Pretest(Winter,2015) Pos4est(Fall,2015)

Controlboys(N=230)

Controlgirls(N=212)

ABRAboys(N=136)

ABRAgirls(N=171)

GRADEscores

ABRA Control

Girls(N=171) Boys(N=136) Girls(N=212) Boys(N=230)

Post Pre Post Pre Post Pre Post Pre

TotalTest 70.53 60.13 62.53 58.21 58.85 60.6 57.36 58.92

StandardDeviation 16.20 16.48 18.63 16.79 16.13 15.04 15.73 14.12

ABRA2015Study

5

ThetestsfortheeffectofABRAonEnglish,Math,ScienceandSocialStudiesexamscoresshowedthedifferencebetweenthegroupswasstatisticallysignificant(p<0.001).ThestatisticssummarizedinTable3andillustratedinFigure3showthatstudentsintheABRAgroupperformedhigherthancontrolstudentsoneachofthefourcoreexams.Table3.CoreexamsresultsforABRAandcontrolclasses(adjustedmeans,andstandarderrorvalues)

1Adjustedmeansforpost-testscoresandstandarderrorcalculatedinthemodel.GRADEpre-testcovariatescoresmodelwereevaluatedat58.18.

Coresubjectexams ABRAMeans1(N=149)

ControlMeans1(N=315)

English 85.86 74.03StandardError 1.34 0.91Mathematics 77.40 68.31StandardError 1.58 1.08Science 86.39 68.87StandardError 1.23 0.84SocialStudies 82.25 65.14StandardError 1.20 0.83

ABRA2015Study

6

Figure3.KenyaEndofYearExaminationResultsbySubjectGradeLevelStudentgradeleveldidnotlessentheeffectsofABRA.StudentsinGrades1,2&3allbenefitedfromABRAregardlessofwhethertheyhadusedABRAinthepreviousschoolyear.Table4andFigure4allowonetofollowhowthemeanGRADETotalTestscoreschangedfrompre-topost-testbygradeinABRAandcontrolgroups.AsexpectedthehighestincreaseisnoticeableforABRAstudentsingrades2and3asrespectively23%and85%ofstudentsinthesegradeshadbeenexposedtoABRAsince2014for1ormoreyears.Table4.GRADETotalTestMeanScoresbyGradeLevel

Difference between the groups: Core end-of-year 2015 exams

10

20

30

40

50

60

70

80

90

English Socialscience Mathema;cs Science

Control(N=315)

ABRA(N=149)

GRADEscores

ABRA ControlGrade1(N=83)

Grade2(N=124)

Grade3(N=100)

Grade1(N=98)

Grade2(N=151)

Grade3(N=193)

Post Pre Post Pre Post Pre Post Pre Post Pre Post Pre

TotalTest 47.58 44.01 69.85 59.6 79.54 71.56 47.37 50.93 51.79 57.46 68.43 65.96

StandardDeviation 14.38 14.95 12.12 13.33 11.57 9.93 8.97 12.26 13.05 14.63 14.53 13.39

ABRA2015Study

7

Figure4.ResultsbyStudentGradeLevelTeacherExperienceWealsoexploredwhetherthereweredifferencesamongteachersbasedonyearsofexperiencesusingABRA.Becauseofinitialtrainingandfollow-upsupportaswellasin-classexperienceoveroneormoreyears,wedidnotexpectthatnoviceuserswouldhavelowereffectscomparedtoexperiencedusers.However,wewereconcernedthatiftherewerenoveltyeffectstheywoulddissipateovertime.However,asFigure5shows,therewerenodecrementsinstudentlearninglinkedtoteacherexperience.

Difference between the groups on GRADE(Total Test) post-test scores: grade level

20

30

40

50

60

70

grade1(Ncontrol=98;NAbra=83)

grade2(Ncontrol=151;NAbra=124)

grade3(Ncontrol=193;NAbra=100)

Control

ABRA

ABRA2015Study

8

Figure5.YearsofTeacherExperienceandABRAEffectsImplications

Theseremarkableresultsareatestamenttotheevidence-baseddesignofABRAandqualityteacherimplementationintheauthenticcontextoftheirschools.ABRAisnotintendedtosupplementclassroominstruction,butinsteadshouldbeintegratedintoclassroominstruction.ThetrainingandsupportprovidedbytheABRAcoordinatorsandambassadorscoupledwithteacherswillingtoworktoimproveteachingarewhatmadethedifference.

SeveralotherimprovementsshouldmaketheefficacyofABRAevenstrongerinyearstocome.First,weexpandedandimprovedourteacherprofessionaldevelopmentresourcesandassembledthemtogetherinTheLearningToolkitTeachersGuide:KenyaEdition.Second,toenhancefurtherstudentfluencyandcomprehensionskills,wedevelopedREADS,afreelyavailabledigitalrepositoryofhundredsofbooksandstoriesinmultiplelanguagesincludingEnglishandKiswahili.READSfitseasilyonaUSBkeyandmeansnoKenyanchildwilllackengagingreadingmaterialtopracticeandhonetheirliteracyskills.Third,wemorecompletelymappedABRAactivitiesandstoriesontotheKenyancurriculumsothatteacherswillfindintegratingABRAintotheirlessonsespeciallyeasy.Fourth,wetackledthechallengeoflargeclasssizesandtherelianceonfrontalteachingbyemphasizingcooperativesmallgrouptechniquesduringABRAlessons.Finally,to

Change in GRADE(Total Test) scores: teacher experience with ABRA

50

55

60

65

70

75

Pretest(Winter,2015) Pretest(Fall,2015)

Oneyear(N=68,grades1,and3)

Twoyears(N=157,grades1,2,and3)

Threeyearsandmore(N=68,grades2and3)

ABRA2015Study

9

complementgender-balancedstoriesandactivitiesinABRA,weofferedsupportmaterialstoteacherstohelpthemadequatelycultivategenderequalityintheirclassrooms.

WeareexcitedaboutthelatestABRAresearchandhopeyouaretoo.Wethinkwe’vetakenanothersteptocreatinganationofreadersinKenyaandbeyond.