Working Paper No. 475

Curriculum Change and Early Learning: An Evaluation of an Activity Based Learning Program in Karnataka, India

by

Kallan Gowda

Anjini Kochar

Closepet Nagabhushana

N. Raghunathan

June 2013

Stanford University John A. and Cynthia Fry Gunn Building

366 Galvez Street | Stanford, CA | 94305-6015

Curriculum Change and Early Learning: An Evaluation of an Activity Based Learning Program in Karnataka, India

Kallan Gowda*

Anjini Kochar+

Closepet Nagabhushana*

N. Raghunathan*

June 2013

Abstract

Despite significant and increasing resources spent on primary schooling in India, improvements in learning have been difficult to achieve. Seeking to redress the system, many state governments have adopted a more flexible learning strategy that allows students to learn at their own pace and accommodates differences in learning levels and abilities within a classroom. Even though such strategies, referred to as Activity Based Learning (ABL), are growing in popularity, there are few rigorous evaluations of their impact on learning. If proven effective, these pedagogical changes hold the promise of improving learning outcomes without significantly enhancing the resources devoted to schooling. The evaluation of such methods is therefore of significant importance, particularly in economies where the ability to significantly increase the resources devoted to schooling may be limited. This paper evaluates one such program, the “Nali Kali” program adopted in the South Indian state of Karnataka, exploiting the phasing in of the program over cohorts and schools to identify its effects.

Keywords: India, Primary schooling, Activity based learning. JEL Classification No.: I21, I28.

* Catalyst Management Services, India + Stanford University This research was generously funded by The William and Flora Hewlett Foundation’s Quality Education in Developing Countries Program, and by the Azim Premji Foundation. It has benefitted from comments and input by Namita Gupta, Ward Heneveld, Lynn Murphy, Chloe O’Gara, Dana Schmidt, and Rishikish Shanker. All errors are those of the authors.

1  

1. Introduction

Despite significant and increasing resources spent on primary schooling in India, improvements in learning have been difficult to achieve. Students score very poorly on grade-specific tests (Assessment Survey Evaluation Research Centre 2011), with the majority failing to achieve expected competencies. Many believe that low levels of learning reflect characteristics of the Indian schooling environment: considerable variance in the (initial) ability levels of students and a corresponding variation in their ability to learn; a high level of student absenteeism that fragments the school year and considerably reduces its length; and the prevalence of multi-grade classrooms that further reduce the time that the teacher can devote to students of any given grade. Such an environment is particularly ill-suited to the traditional teacher- and textbook-centric instructional method that requires teachers to work through a grade-specific curriculum over the course of the school year, without making allowances for differences in students’ learning abilities and levels. Seeking to redress this, many state governments have adopted a more flexible learning strategy that allows students to learn at their own pace and accommodates differences in learning levels and abilities within a classroom. Even though such strategies, referred to as Activity Based Learning (ABL), are growing in popularity, there are few rigorous evaluations of their impact on learning. If proven effective, these pedagogical changes hold the promise of improving learning outcomes without significantly enhancing the resources devoted to schooling. The evaluation of such methods is therefore of significant importance, particularly in economies where the ability to significantly increase the resources devoted to schooling may be limited.

This paper evaluates one such program, the “Nali Kali” program adopted

in the South Indian state of Karnataka, exploiting the phasing in of the program over cohorts and schools to identify its effects. The potential for credibly evaluating programs that are phased in over several years has been recognized by many researchers. In most cases, the phase-in occurs geographically, with the spread of the program varying across regions and years. In such cases, if the regions or schools to receive the program first are systematically selected on the basis of some specific criteria, an analysis based on a comparison of outcomes across early and late beneficiaries will likely generate biased estimates of the effect of the program. A program that is phased in by cohort, such as the Nali Kali

2  

program, lacks this source of bias but is subject to the criticism that exposure to the program varies in a cross-section of data only by cohort, and, for any given cohort, varies over time with the grade level. That is, identification comes from a cohort-year effect, so that any identified effects of the program may be a consequence of schooling factors that are cohort specific, but vary across schooling years. In this particular program, however, its early piloting in all small schools also generates variation in years of exposure within a cohort. It is therefore possible to identify the effect of years of exposure to the program, while simultaneously controlling for cohort, grade and year effects. This considerably strengthens identification.

The remainder of this paper is structured as follows. In Section 2, we

briefly discuss the context in which this study takes place and provide details of the program. Section 3 describes the survey area and the data. The econometric methodology is specified in Section 4, while Section 5 contains results. The last section concludes.

2. Learning Levels, Instructional Methods and the Nali Kali Approach 2.1 Learning Levels in India

Learning levels in India remain very low, despite the significant resources

devoted by the Government to primary schooling. Data from the Annual Status of Education Report 2011 (Assessment Survey Evaluation Research Centre 2011) reveal that, in rural India, 48% of students in grade 5 can read only at the level of grade 2, while 24% can read only at a grade 1 level. Similarly, 24% of grade 5 students can recognize two digit numbers, but cannot do subtraction or division and 34% cannot perform simple division.

One reason for low levels of learning may be India’s historical adherence

to a common curriculum for each grade. This pedagogical approach meant that all children within a grade were uniformly taught, using the same textbooks and methods. It also led to a rigid classroom environment, with teachers required to ensure completion of the textbook material within the school year, without regard

3  

to students’ comprehension of the material. This environment did not accommodate the very low (initial) learning levels of students, many of who were first-generation students. For those starting at a level below the standard expected from incoming students, catching up was difficult. Teachers, required to implement a centrally devised plan that specified what was to be taught at different stages of the school year, were not in a position to focus on the low learning of a segment of the class.

A “common standards” approach can have particularly detrimental effects

when applied in the multi-grade classroom environment that characterizes so many of India’s primary schools. Teachers required to teach a grade-specific curriculum in a multi-grade environment must, per force, divide their instructional time amongst the different grades in the classroom. This considerably reduces classroom time devoted to any particular cohort of students, with a concomitant reduction in learning. The students who are most in need of instruction are likely to suffer the most under this system.

2.3 Activity Based Learning

Recognizing that pedagogical methods can significantly enhance learning, different states have experimented with a wide variety of approaches since the 1990s. Many of them involve Activity Based Learning (ABL), a pedagogical approach that emphasizes skill acquisition through activities, and promotes a flexible teaching style that can accommodate differences in children's learning levels and capabilities.1 Instead of textbook- and teacher-centered learning, an ABL classroom generally students divided into several groups, each at different stages of learning and following different learning strategies requiring different levels of teacher involvement.2 The learning process is broken into small steps that collectively form a “learning ladder.” Students work at their own pace up this “ladder,” an approach that allows students who have missed a few school days to step back into the classroom and pick up from where they left off.

                                                            1 In India, ABL was pioneered by the Rishi Valley School and the Rishi Valley Institute for Educational Research (RIVER). 2 One group may require the full attention of the teacher, while others may allow for more peer learning or for self-directed learning by more advanced students.

4  

2.4 The Karnataka State Government’s Nali Kali Program

The program was first introduced in the state on an experimental basis in H.D. Kote Taluka of Mysore District in 1995-96, and subsequently extended to all rural areas of the district in 1997-98.3 In 1999-2000, it spread further to schools in blocks covered by the UNDP’s Janshala and the World Bank’s DPEP programs. In 2004-05, it was introduced in small schools in an additional 8 blocks of the state. In 2007-08, the program was introduced in grades 1 and 2 of all schools in the state with a total enrollment of less than 30 students (as per available records). Finally, commencing June 2009, the program was implemented in grades 1 and 2 of all Kannada medium schools, including schools with enrollments exceeding 30. The following year, the program was extended to included students in grade 3.

As in other Activity Based Learning models, an important component of

the Nali Kali model is the reorganization of the curriculum into small manageable units, with different units sequenced into a comprehensive “learning ladder.” Activities and learning material are prepared for each step of this ladder, and children within a classroom are allowed to progress at their own pace, and only after they have mastered the material at their current level. Learning materials are provided for each level. For example, instruction at the grade 2 level is supported by a set of 50 reading books, numbered 1 to 50, each at a progressively higher level of difficulty.

Children are grouped within the classroom in a manner that allows for

maximum cross-learning and different levels of supervision and instruction by the teacher. Teachers are expected to focus their efforts on students at lower levels of learning, and ensure that all children reach a stipulated level of learning by the end of the grade. Because children are organized in groups that differ in their learning levels, this approach is thought to be particularly well-suited for the multi-grade environment. The process also removes traditional examination-based assessment procedures, with assessment following naturally by recording each child’s progress up the learning ladder. Children’s active participation is sought in the learning process: they are asked to identify their position in the learning

                                                            3 With the exception of Mysore Urban Taluka which had a high percentage of private schools

5  

sequence and to plot their own progress, and their work is prominently displayed in the classroom.

2.5 Evidence on Activity Based Learning Programs Since ABL programs have only recently been introduced on a large scale, there are relatively few studies that evaluate their performance. Available studies primarily provide a qualitative assessment of the implementation of the program, with little information on its effectiveness in improving ability levels (SchoolScape and Sarva Shiksha Abhiyan, Government of Tamil Nadu, 2009; Macchiwalla). The SchoolScape-SSA study did assess learning achievement by comparing test scores from baseline and end-line tests (for students in grades 2 and 4) conducted at the beginning and end of the year that ABL was universally introduced in Tamil Nadu (2007). These scores suggested a significant improvement in average learning levels over the course of the year, and a reduction in achievement gaps across gender, location (rural and urban) and social groups. However, the absence of any kind of control sample makes it impossible to ascribe the improvement in learning over the course of the year to ABL; one would expect some level of improvement, even in the absence of ABL.

3.TheSurveyAreaandSurveyData3.1SurveyDataandSample

We use survey data from the South Indian state of Karnataka, a state thatranks 7th of the 16 major states of the country in terms of educationalperformance. The survey covers 11 districts that span the entire state,covering all geographical zones. Karnataka uses a “cluster” approach forteachertrainingandschoolmonitoringpurposes,witheachclustercoveringapproximately 15‐20 schools.Our surveyprovidesdata on60 clusters andapproximately180schools.Forsamplingpurposes,thenumberofclustersineach district was determined in proportion to the district population.Clusters within a district were then selected using systematic randomsampling.Selectedclusters(ineachdistrict)wererandomlyallocatedtoone

6  

offourgroups.Threeofthosefourgroupsweresetasideforevaluationsofexperimentsto improvethe functioningofschoolmanagementcommittees.Thelastgroup,intendedtoserveasacontrolfortheseevaluations,formsthesample for thisstudyonNaliKali.Withineachcluster,villagegovernments(GramPanchayats)were randomly selected,withone government selectedin small clusters and two in larger ones. In each Gram Panchayat, wesurveyed the smallest and the largest school, and a third school randomlyselectedfromtheremainingsetofschools.

The survey was initiated in 2009‐10, with the last round of data

collectionoccurringinthe2012‐2013schoolyear.Surveymodulesincludeda standard school survey, conducted in 2009‐10, 2010‐11 and 2012‐13.Students were tested at the beginning (approximately August) and end(approximately February) of each of three school years (2010‐11 to 2012‐13). The analysis of this paper is confined to test scores from the August2010,August2011,August2012andFebruary2013surveys.WedonotusedatafromtheearlierFebruaryrounds,becauseofimplementationdifficultieswhich compromised the quality of this round of testing data. The Augustroundsaretakentoreflectanend‐lineforthepreviousschoolyear.Thus,forexample, theAugust2010dataare taken tobe reflectiveof learning in the2009‐10 school year. The February 2013 data serve as an end‐line for the2012‐13schoolyear,thoughwealsoprovideasetofresultsbasedonjusttheAugustdata,ignoringthedataobtainedinthelastFebruaryround.

Withthislastroundincluded,end‐linedataisavailableforfourschool

years,andforfourcohorts.Thefirstcohortisthegroupofstudentsingrade3in2009‐10,whilethefourthcohortisthegroupenteringgrade1in2010‐11.Thefirstcohortofstudentstransitionedtohigherprimaryschool(grade6)inthelastsurveyyear.Forthoseenrolledinlowerprimaryschoolsonly,thismeantatransitiontoadifferentschool.Weworkedwithschoolteachersand other administrators to determine the students of this set who haddroppedoutofschoolandthosethathadcontinuedon.For thosewhohadcontinuedtheirschooling,weidentifiedthehigherprimaryschool inwhichtheywere enrolled and tested them,despite theirmovement to adifferentschool.

7  

Thetestsusedinthissurvey,verysimilartothoseusedintheAnnualStatusofEducationReport(ASER),4weredevelopedbyanexternalagency,andallowanassessmentofstudentlearningrelativetocompetenciesforallgradescompletedbythestudent.Forexample,grade3studentsweretestedintheirknowledgeofgrade1competencies,grade2competenciesandgrade3 competencies. The summary statistics reported in this paper and testregressions are based on test scores on sections which evaluatecompetenciesinthegradethatthestudentisenrolledin,aswellasonelowergrade.Thus,end‐linetestsforstudentsingrade3arepercentagescoresonquestionsthattestgrade2andgrade3competencies.

Great carewas taken toensure that the testswerecomparableover

time.Thiswasdonebyretainingthesametypeofquestionsineachtest,butonly changing content. Mathematics tests, for examples, changed over theyears only in the numbers in each question, not in the type of question(replacing6+4,forexample,with7+3).Similarly,thecomplexityoflanguagetestsremainedthesamefromyeartoyear.Testswereadministeredbyourfield investigators without any supervision by school teachers, and onunannounced school visits. Permission to conduct testswas obtained fromthestategovernment.

Asignificantcontributionofthisstudyisanassessmentofchildren’s

non‐cognitiveskillsinspecificareas.Theagencythatdesignedthetestshasspecificexpertiseintestingnon‐cognitiveskillsandhaspreviouslydesignedtests for several schools and the major non‐government organizationsinvolved in education. Tests were designed to evaluate the students inleadership, social and communication skills. The tests involved individualand groupobservationsof childrenparticipating in twodifferent activities,over a 2 ½ hour period. Each group of students was assessed by twoexaminersspeciallytrainedforthepurpose.Asanexample,onesuchactivityinvolved students deciding the end of a story that was read to them. Theactivitywasconducted in threeparts.First, children listed to thestoryandwereaskedtothinkaboutresponsestoasetofquestions.Second,children

                                                            4 This is a highly reputed test conducted throughout India by a non-governmental agency, Pratham.

8  

weredividedintogroupswithamaximumsizeof10.Eachchildwastosharehis/her decision with the entire group. The children were then to have agroup discussion and arrive at a joint decision regarding the conclusion ofthestory.Onconclusionofgroupdiscussions,eachstudenthadtoapproachtheexaminersindividuallyandexplaintheoutcomeofthegroupdiscussions,anydifficultiesinthegroupprocess,whethertheyagreedwiththeoutcomeorhowtheywouldhavechosentoendthestory.Third,childrenweregivenachart paper and a set of crayons, and were asked to work together toconceiveanddrawanyscenefromthestory.Thegroupwasaskedtoensurethatallchildrenwereinvolvedintheactivity.Assessorsgradedstudentson10questionsforeachskill.Foreachquestion, theexaminergave1pointtothestudentifheorshedemonstratedthedescribedbehavior.Thus,eachskillwasevaluatedona10pointscaleforeachofthetwoactivities.5

3.2SummaryStatisticsonTestScores

Summarytestscoresarepresentedintables1,bycohortandgrade.Thedatarevealadecliningtrendinlearning,relativetograde‐specificstandards,fromone grade to the next for all cohorts, particularly in language.6 That is,students are able tomastermore of the curriculum in earlier grades thantheyareinhighergrades.Thoughstudentstestedattheendofgrades1and2 do well relative to stipulated standards for these grades, performancedrops off significantly at higher grades. For example, cohort 2 studentsreportedanaveragetestscoreof58%inlanguageand55%inmathematicsat theendofgrade2.Thesescoresdroppedto44%and49%at theendofgrade3, andwereas lowas36%and41%, for languageandmathematics,                                                            5 For example, in evaluating confidence, examiners had to grade students on the following: (1) did the student make eye contact with members in the audience? (2) was the student’s body language confident in that the student stood erect with an open posture (3) Was the student composed and poised while addressing the group or nervous and fidgety (4) did the student retain poise even if he/she were unable to express themselves well (5) Did the student state ideas without being worried about how they would be received? (6) Did the student explain ideas or continue participating even when ideas were not well received (7) Was the student comfortable with trying all proposed activities without worrying about failure (8) Did the student suggest one item on his/her own? (9) Did the student make decisions within the time provided (10) did the student give reasons for actions and positions? 6 As previously noted, reported test scores are for percentage scores on competencies for the student’s current grade and the previous grade.

9  

respectively,attheendofgrade5.Thedeclineintestscoresathighergradessuggeststheimportanceofallowingforgrade‐specificvariationintestscoresintheregressionanalysis.

The data also suggest significant cohort effects, revealing a generalimprovement in test scores for each successive cohort. Test scores for thefirst cohort, when in grade 3, averaged 40% and 39% in language andmathematicsrespectively.Averagegrade3testscoresimprovedto52%and51%forstudentsfromcohort4.

This trend may well represent real improvements in learning overtime, a consequence of the government’s increased and continuousinvestments in primary schools as well as of programs such as Nali Kali.Thereisalsoapossibility,however,thatyear‐to‐yearvariationintestscoresreflect variation in implementation, despite the fact that the content of thetest (for each grade) was kept the same from year to year.7 Improvedperformance over timemay also be the consequence of students’ growingfamiliaritywiththetests.Asdetailedlater,toensurethatthesepossibilitiesdonotaffectregressionresults,allregressionsreportedinthispapercontrolforyear,gradeandcohorteffects,aswellasfortimetrends.

Therearenobenchmarksforthenon‐cognitivetestscores,soitisnotpossibletocomparestudentsinthisregardtothoseinotherstatesorotherareas. Average test scores area generally highest for communication andconfidence skills, and lowest for leadership skills. Unlike test scores forlanguage andmathematics, however, all non‐cognitive test scores improveover time, as students move from one grade to the next. Particularly incommunicationandconfidence,thebiggestimprovementappearstooccuratearlygrades,forexample,inthetransitionfromgrade2to3,taperingoffasstudents enter grade 5. There is also less evidence of any systematic yeareffectinnon‐cognitiveskills.

                                                            7 Field personnel typically read out examples at the start of the test and addressed any questions, prior to the implementation of the test. Survey teams, however, varied from year to year, and this might give rise to year-specific differences in test scores, if implementation methods varied. The supervising team, however, remained the same across the survey, so that there is little reason to suspect systematic differences in implementation.

10  

3.3 Testscoreresultsbyquartileoftheabilitydistribution Declining performance in tests over time raises important questionsconcerningwhichstudentsarefallingbehind.Weexaminethisgraphicallyinfigures 1 and 2, identifying students by cohort and their initial (cohort‐specific)learningquartile,basedontestscoresinthebaseyear(August2010forcohorts1to3,andAugust2011forcohort4),separatelyforlanguageandmathematics.

These graphs reveal a significant compression of the test scoredistribution, for both language and mathematics over time; there isconsiderable variation in test scores in the initial survey year (for eachcohort),avariationthatissignificantlyreducedasstudentsprogressthroughgrades. The fact that the initial testing date for the fourth cohort (August2011) differs from that of the first three cohorts (August 2010), and thatinitial test scores for this cohort are still characterized by considerablevariation suggests that this compression is not caused by any anomaly intestinginthefirstyear.

Thestrikingfeatureofthesegraphsistheirrevelationthatthedecline

in average test scoresover time, relative to grade‐specific competencies, iscaused by a decline in performance of students at the top quartiles of theacademicdistribution. Students at thebottomof thedistributionappear toimprovetheirperformanceovertime.Whiletheexplanationforthispatternis far from clear, it may reflect teachers progressively concentrating onminimum learning standards as students progress through grades,eliminatingthepossibilityforbetterstudentstocontinuouslyimprove.

DidtheintroductionofNaliKalihaveanyeffectonthesetrends?The

graphs for cohorts 3 and 4 reflect learning in the early grades, under NaliKali.However, theycannotbecompared to thegraphs for cohorts1and2,since the test scores for different cohorts pertain to different grades. Forexample, thegraphs for cohort1demonstrate learning fromgrades3 to6,while those for cohort 4 span grades 1 through 3. But, while the graphscannotbeused tocompare learningunderNaliKali to thatachievedunder

11  

older instructional methods, they do provide information on whether thedeclineinlearningofstudentsinthetopquartilesaswellasthecompressioninthedistributionarealsoobservedunderNaliKali.Thegraphsforcohorts3and4 suggest that theyare.Thus, even students in cohort4, forwhomweonly consider data for grades 1 through 3, reveal this same pattern of adecline in average learning levels caused primarily by a sharp drop‐off inperformancebetweengrades1and2forstudentswhowereatthetopoftheabilitydistributionintheirfirstyear.

It shouldbekept inmind that thesegraphsdemonstrate learningat

gradelevel.Theysuggestthat,progressively,studentsknowlessandlessofwhat they are supposed to know at any specific grade. This does not,however, implythatthereisno learningoccurringinschools;studentsmaywellbelearningmoreeachyearrelativetoastaticbenchmark,eventhoughthey do badly relative to progressively higher grade‐level competencies.Becauseeachyear’stestsincludedquestionsatdifferentgrades,weareableto infer improvements in absolute learning levels (rather than learningrelative to grade standards). Improvements in learning levels relative to aconstantbenchmarkaregraphedinfigure3,forlanguageandmathematics,for the first two cohorts. For cohort 1, we evaluate learning at differentgrades,relativetothegrade4competencies.Grade3competenciesareusedtoevaluate the learningachievedbystudents incohort2. Inboth languageand mathematics, knowledge improves over time. The failure of schools,then, is to teach tostandard; thoughstudentsdobetteratbasic levelsovertime, they increasingly fall short of the rising standards that are expectedoverthecourseofprogressionthroughschool.

This failure, forgrades taughtprimarilyunderNaliKali,mayreflect

thewayinwhichtheprogramisimplemented.Whilestudentsaresupposedtolearnattheirownpace,theleveloflearningexpectedtobeachievedattheendofeachgradeisclearlystipulated.Indeed,thestepsandmilestonesthatare to be achieved at the end of eachmonth are clearly communicated toteachers. Thismeans that all students generally start amonth at the samelevel.And,ifastudentdoesprogressfaster,theyarelikelytobe“heldback,”ratherthanallowedtoprogresstothesetofstepsrequiredtobecoveredin

12  

thenextmonth.Thatis,thoughtheprogramdoesbreaklearningdownintosteps, andhencedoesabetter job inensuring thatall thebasicmaterial iscovered,it is less likelytoallowadvancedstudentstoproceedattheirownpace.

We confirmed this through an examination of the registers

maintainedineachschoolthatcharttheprogressofeachstudent,onadailybasis, in each subject. A page from one such book is displayed in figure 3.Thereare three rowsofdataentered for each student, recording the stepsachieved in language, mathematics and environmental sciences (EVS)respectively.Thus,thefirstthreerowsreporttestscoresinthesesubjectsforthe first student, the next three rows report test scores for the secondstudent and so forth. The data reveal that all students start themonth atapproximately the same level; there is little variation in scores at thebeginningofthemonth.And,thefewstudentswhodostartatahigherlevelappeartoproceedmoreslowly,sothattheyendupatthesamelevelasotherstudents by the end of the month. For example, student 2 started at asignificantly higher level for mathematics (at step 11, row 5 of the table,compared to the modal value of 6) than other students. However, herprogressionthroughthemonthwasslower,sothatbyday17,shewasatthesame levelasotherstudents in theclass.8Thetablealsoclearlyshowsthattheexpectedmilestoneof18 forEVS targeted for thismonthwasachievedrelativelyearly.Studentswerethenkeptatthismilestoneuntiltheendofthemonth.

It is also worth noting that it was expected that students would

completestep49inlanguageand60inmathematicsattheendofthemonth.Thus,thelevelcompletedbymoststudentsattheendofthemonth–43forlanguageand53 formathematics– fell shortof expected levels, suggestingslowlearninginthesystemforallstudentsrelativetowhatwasexpected.

                                                            8 Multiple entries in any given cell record all the steps that were covered by the student in any given day. Thus, for example, the entry of “14,15,16” for student 1 in the first row on day 12 reveals that she completed these three steps in language on that day.

13  

A child’s progress, as reflected in entries in school ledgers,may notreflect his or her actual learning level. However, under Nali Kali, becauseeachstudentisgivenaworksheetspecifictothelearningstepthattheyareon, learning is more likely to reflect recorded progress; children are notgiven worksheets for advanced steps and hence cannot move beyond thetargets set for eachmonth. Our independent tests, in fact, confirmwhat isrevealed in the registers: there is a compression of learning abilities, withbetterstudentsbeingheldbacktothelevelofotherstudentsintheclass.

4. EconometricFramework

4.1 Identificationofyearsofexposuretotheprogram

We identify the effect of the Nali Kali program on learning byexploiting the phasing in of the program across grades over a two yearperiod, as well as the within‐cohort variation generated by piloting theprograminsmallschoolsin2007‐08.

To establish the basis of identification in this paper, tables 2 and 3document years of exposure toNali Kali under the assumption that itwasimplemented as planned, and so introduced in grades 1 and 2 of smallschoolsin2007‐09anduniversallyin2009‐10,withanextensiontograde3in 2010‐11. Table 2 reveals the variation in years of exposure under thisphase‐inprogramfor4cohortsovertheirprimaryschoolingyears,withtheoldest(cohort1)being ingrade6 in the lastsurveyyear,2012‐13.Table3reports this same information, but also shows the variation within eachsurvey year. The second cohort of students (grade 5 in 2012‐13) in largeschools,forexample,wouldhavecompletedoneyearofexposuretoNaliKaliat theendof the2009‐10 school year,while, in the sameyear, students ingrade3intheseschoolshadneverreceivedanyinstructioninNaliKali.

The tables make clear the sources of identification of the effect ofyearsof exposure to theprogramon learning.Consider, first, identificationrelying just on the large school sample and hence ignoring within cohort

14  

variationinexposuretotheprogramForcohort2studentsinlargeschools,thedurationofexposuretotheprogramvariesbetweengrades2andgrades3. If data were only available for this cohort of students, the variation inexposurewouldbeindistinguishablefromagradeeffect,i.e.,fromchangesinlearning that may occur between grades 2 and 3 for a variety of reasons.Addingonanadditionalcohortof students, say those incohort3,anddatafor at least two years, allows us to observe different cohorts in the samegrade,andhenceenablescontrolsforcohort,gradeandyeareffects.

However, table 2 reveals that the increase in years of exposure forcohorts2and3inlargeschoolsfollowsasimilartimeprofile:Itincreasesbyone year between grades 2 and 3, and then stays at the same level.Identification using data on just these two cohorts, controlling for cohortdifferences, is thus driven by a functional form assumption regardinglearning over time. Specifically, it requires the assumption that thenormallearning pattern over grades does not follow this samepattern of years ofexposuretoNaliKali.ThediscontinuouspatternofyearsofexposuretoNaliKaliovertimesuggeststhatthisassumptionislikelytobemet.Nevertheless,itremainstruethatanalysisbasedjustondatafromlargeschoolsidentifiesthe effect of years of exposure toNaliKali basedprimarily on a functionalformassumptionregardingthepatternoflearningovergrades.

Dataonacohortwithadifferenttimeprofileofvariationinexposure

to Nali Kali, cohort 1, significantly aids identification. Becausemembers ofthis cohort in large schoolswere not exposed toNali Kali, it is possible toallow for a time trend to capture any systematic improvement in learningover time, and even to allow the time trend to vary by cohort, and stillestimatetheeffectofyearsofexposuretoNaliKalionlearning.

It is the variation in years of exposure to Nali Kali within a cohort,

causedbytheearlyadoptionofNaliKaliinsmallschoolsthat,incombinationwiththesecohort‐specifictrendsinyearsofexposuretoNaliKalicausedbythephasing inof theprogram to thirdgrade studentsover time, generatescredible identificationof theprogram.The fact that thedifferencebetweenthe small schools thatwereeligible for theprogramand the larger schools

15  

thatwerenotonlygeneratedvariation inyearsofexposure toNaliKali forcohorts1and2,butnotsubsequentcohorts,makes itpossible to includea“eligible school” dummy in regressions, and still identify thewithin‐cohortvariationinexposure.

DataonyearsofexposuretoNaliKaliisobtainedfromschoolrecords

of when the program was first introduced, rather than eligibility rules.Though the program was to have been introduced in all small schools in2007‐08(excludingschoolsinwhichthelanguageofinstructionwasUrduorMarathi), our survey data reveal that it was in use in only 65% of smallschoolsin2007‐09.Weclassifysmallschoolsbyschool‐reportedenrollmentdatafor2007‐08,andtheremaywellbesomediscrepancyinthedataweuseand those used by the Government. Lacking an administrative list on theschoolsinwhichitwasintroducedin2007‐08,wegobyschoolrecordsofthedateofimplementation.

Despite theapproximately35%ofsmallschoolswithoutNaliKali in2007‐08, thevariation in implementationofNaliKaliacrossschools in thisyearwasclearlybasedonschoolsize,asperthepolicy.Oflargeschools,withenrollmentsinexcessof30,90%ofschoolsreportedtheuseofthesystem.This suggests that any bias introduced by the selection of early adoptingschools can be eliminated by including quadratics in (2007) schoolenrollment amongst the regressors, as well as the eligible school dummyvariable that takes the value 1 if enrollments in 2007 were less than 30.Additionally, we also include quadratics in enrollment in primary grades(grades1to5),aswellasinthestudent’scohortsize,asmeasuredbycohort‐specificgrade1enrollmentdata.

Since identificationreliespartlyon thevariation inexposure toNaliKaliacrosssmallandlargeschools,ourmethodologyidentifiesa“local”effectthatappliesspecificallyforschoolsthatwouldhavehadadifferenceinyearsof exposure had their enrollment been higher. Should there be significantheterogeneity in the effects of the program on learning across schoolsdistinguishedbysize,thenidentificationofa localtreatmenteffectrequiresrestrictingour sample to relatively small schools. Thus, in regressions that

16  

exploit thewithincohortvariation for identification,werestrict thesampleto schools with fewer teachers than grades,9 that must therefore utilizemulti‐gradeinstruction.

Our overallmethodology is similar to that used by Duflo (2001) tostudy the effect of a school expansion program in Indonesia on years ofschooling.As in theproposedstudy, theeffectof theavailabilityof schoolswasevaluatedbyexploitingtheintensitytotreatmentofdifferentcohortsofstudents.Theproposedstudy,however, improvesonDuflo’s study throughtheavailabilityofchild‐leveldataonlearning.

Our baseline regression therefore examines the effect of years of

exposure to Nali Kali (yrs_nk) on test scores, A, in regressions that alsocontrol for cohort (μc), grade (μg), year (μy) and district (μd) fixed effects.Additionalregressors includechild‐specificcharacteristics (gender,ageandcaste,X),andcohortcharacteristics (Zc, initialcohortsizeand itssquare,aswell as the initial proportion from scheduled castes and tribes, with bothvariables constructedusingenrollmentdata foreachcohortwhen ingrade1).School‐levelinfluencesonlearning(S),includetotalschoolenrollmentin2007‐08 and its square, primary school enrollments and its square, theproportionof students from scheduled castes and tribes in this same year,the number of classrooms in the school, availability of toilets anddrinkingwater,distancetotheclusterandblockoffice,andtheproportionofteacherswhoaremaleandfromscheduledcastesandtribes.School‐levelregressorsalso include a dummy variable for whether the school is a lower primaryschoolorahigherprimaryschool,andforwhetherit is locatedinthemainvillageorahamlet.Finally,ourbaseregressionincludesatimetrendtoallowforanytrendineitherdrop‐outratesorinlearning.Forstudentiofcohortc,gradeg,enrolledinschools,andobservedinyeart,theregressionequationis:

                                                            9 With the number of teachers being calculated according to defined norms. The regression sample therefore is restricted to lower primary schools with an enrollment of 120 and higher primary schools with an enrollment of 210.

17  

(1) tcsticstcgstogstic SZXNKyrsA 4321 )_(

icgstdygc u

Theregressionsampleused for themainregressionsof thepaper is

restrictedtothesampleofsmallschools,thoughwealsoreportresultsfromregressions that test the sensitivity of results to this restriction. Weadditionally remove test scores for students in grade 1, since tests at thislevelarenotbelievedtobereliable,giventheunfamiliarityofstudentswithtesting. Finally, test scores for students in grade 6 are also removed, sincethey do not add to identification, and hence merely increase inefficiency.Sensitivitytestsalsoexploreanybiascausedbythisexclusion.

To allow for correlations in test scores amongst students within a

school, all reported standard errors allow for clustering at the level of theschool.

4.2 SampleselectionbiasAsinanyanalysisoftestscores,biascausedbystudentabsenteeismonthetest date is an issue, particularly in developing economies whereabsenteeismisasignificantconcern.Inaddition,therelianceoftheanalysisondataovertimealsoraisesconcernswithsampleattritionduetodrop‐out.Bothdrop‐outandabsenteeismcombinetocausevariationintheprobabilityofinclusioninthesampleandraisethepotentialforselectionbiasifexposuretoNaliKali iscorrelatedwiththisprobability.This, in turn,couldhappen either because exposure to Nali Kali affects students’ choicesregardingattendanceandenrollment.Alternatively, itmayalsobe thecasethat the trends over time and across schools that identify the variation inyearsofexposuretoNaliKalialsoexistinthedataonsampleparticipation,causingbiasedresults.Aspreviouslynoted, thediscontinuousnatureofthetrend inyearsofNaliKali exposureover time, its variationacross cohorts,and its variation across schools makes it unlikely that such a correlationexists. Nevertheless, this section presents data on drop‐out rates and

18  

absenteeism,aswellasregressionestimatesoftheeffectofexposuretotheprogramontestparticipationprobabilities.

This analysis is facilitated by the detailed data we collected on theenrollmentstatus,ineachyear,ofstudentspresentinourfirstsurvey.Thesedataallowustoidentifythestudentswhohavedroppedoutofschoolaswellasthosewhoareenrolledandattendregularly,butwereabsentonthetestdate.

Table4reportstheproportionofstudents,ineachcohort,whowere

in the survey at the first survey date (August 2010) and still enrolled inschoolinthesecond(2011‐2012)andthirdsurveyyears(2012‐13).Thedataarereportedforallschools,andseparatelyforlargeandsmallschoolswithfewer permissible teachers than grades, with the number of permissibleteachersbasedonthestategovernment’sstudent‐teachernorms.Separatelyreportingstatisticsforsmallandlargeschoolsallowsustoascertainwhetherany observed trends variedby school size, as do years of exposure toNaliKali.

The data suggest that drop‐out rates increase with each grade. Of

studentswhowereinoursampleinthefirstyearofoursurvey,8%ofthoseingrade2 in2010‐11droppedoutby theendof theyear.Thispercentageincreasedto12%forthoseingrade3and14%forthoseingrade4.Dropoutratesincreasesubstantiallyinthetransitiontohigherprimarygrades(grade6).Thus,ofthefirstcohortofstudentswhowereingrade4in2010‐11,only73%were still in school at the closeof the survey,when they shouldhavebeeningrade6.Thispercentagewasrelativelyhigherinlargeschools(81%)compared to small schools (67%), most likely because large schools weremore likely tohavehigherprimarygradesoffered in theschool, so thatnoadditional transportation costs would be entailed for students alreadyregisteredinsuchschoolsintransitioningto6thgrade.

Table5reportsdataon theproportionofenrolledstudents for took

ourtestsineachsurveyyear,bycohortandschoolsize.FortheAugust2010and 2011 tests, these data reflect student absenteeism from school on thetest‐takingdate.However, in the lastsurveyyear,2012‐13,anattemptwas

19  

madetoensuretheavailabilityoftestscoresforallstudenns,byprolongingthe length of time that survey teams stayed in each village. The relativelyhighproportionoftesttakersintheAugust2012andFebruary2013roundsreflectthesuccessoftheseefforts.

Boththedataondrop‐outratesandtest‐takingindicateatrendover

time, but the higher participation rate achieved in the last survey yearsuggestthattheireffectonaverageperformancemaydiffer.Theincreaseinthe drop‐out rate over time, for every cohort, suggests a reduction inreportedaverageabilitybasedontestscores,iflow‐achievingstudentstendtodropoutofschoolinearlyyears.Incontrast,theincreasedsuccessofthesurvey team in ensuring testing of students in the last survey year wouldlower average test scores over time, under a similar assumption that low‐achievingstudentsaremorelikelytobeabsentonanygivenday.

To directly examine whether years of Nali Kali are correlated with

sampleparticipation,table6providesdataonsampleparticipationratesforeachcohort,byyearsofexposure to theprogram.The tablecombinesdatafromsmalland largeschools;students incohort1, forexamplewhoreportyearsofexposuretoNaliKaliarestudentsinsmallschoolsthatwereeligiblefortheprogramin2007‐08.Thedatadonotsuggestanysignificanteffectofyearsofexposureonsampleparticipationrates.

Thisisconfirmedinprobitregressionsoftheyearsofexposureonthe

probabilityofastudenttakingthetest,andhenceontheavailabilityoftestscores for the student in question (table 7). All regressions include cohort,grade, year and district fixed effects, a dummy indicator for whether theschoolwaseligibleforearlyadoptionbasedonits2007enrollment,andthestudentandschool‐specificregressorsdescribedintheprevioussub‐section.The first regression is run on the full sample of schools, Subsequentregressionslimitthesampletosmallschools.Regression2utilizesthesameset of regressors as regression 1, but on this smaller sample. Regression3addsonatimetrendtoallowforthetrendindrop‐outratesandinlearning,while regression 4 allows this trend to vary by cohort. Finally, the last

20  

regression also allows the time trend to vary across eligible (small) andineligibleschools.

All regressions suggest that years of exposure to Nali Kali had no

statistically significant effect on participation in the sample. As previouslynoted, this is not surprising, given thediscontinuous trend in years ofNaliKali foranygivencohort,aswellasthevariationacrosscohortsandacrossschools. The results therefore suggest that any estimated effect of years ofexposure on test scores are unlikely to be biased because of variation insampleparticipationacrossstudentsandovertime.

5. Results5.1 EffectofYearsofExposuretoNaliKalionlanguageandmathematicstestscores

Table 8 provides regression estimates of the effect of Nali Kaliexposure on language test scores, while table 9 does the same formathematics.Thefirstregressionineachtablereportsresultsfromthebasespecificationofequation(1).ExploitingthevariationinyearsofexposuretoNali Kali across cohorts, the second regression replaces the common timetrendwith a cohort‐specific one. Regression 3 allows an interaction of theeligibleschooldummywithatimetrend.Finally,thelastregressioninbothtablesallowsforaverygeneralpatternoflearningovertimeforeachcohort,replacingcohort,gradeandtimetrendswithacohort‐gradeinteraction.Thisregressionallowsfordifferenttimeprofilesforlearningforeachcohort,andstill identifies the effect of years of exposure to the Nali Kali program onlearning. Tests of significance, for the regression as a whole and for theequality of cohort‐specific time trends, are reported at the bottom of eachtable. For language andmathematics test scores, these tests reject thenullhypothesisofequalityoftimetrendsacrosscohorts.

The results for language reveal a significant increase in test scores

whencohort‐specifictimetrendsareincluded(regressions2and3,relative

21  

to regression1).However, allowing for cohort‐grade fixed effects does notsignificantly change results. Our preferred regression, therefore, isregression3,withcohort‐specifictimetrendsaswellasaninteractionoftheeligible school dummy with the time trend. The same pattern is true formathematicstestscores:allowingforcohort‐specifictimetrendsmarginallychangescoefficients,butcohort‐yearinteractionsgenerateresultswhicharenotsignificantlydifferent fromthosethatomit these interactions,replacingthemwithacohort–specifictimetrend.

Theresultsrevealastatisticallysignificanteffectofyearsofexposure

toNaliKalionlanguagetestscores,butnotformathematics.Thecoefficientonyearsofexposure for language tests is statistically significantat the5%level. Theestimates imply thata1 standarddeviation increase inyearsofexposure to Nali Kali would raise language test scores by 0.10 standarddeviationsandMathematicstestscoresby0.06standarddeviations.Thisinturn suggests that one additional year of instruction in Nali Kali increaseslanguage and mathematics test scores by 2.2 and 1.3 percentage pointsrespectively.

5.2 SensitivityAnalysis

Beforewe explore heterogeneity in the estimated effects,we examine thesensitivity of the results to different sample restrictions. Table 10 firstreports regression results from our preferred specification, but with thesampleextendedto include largeschools.Thesecondsetofregressions inthis table reports results from regressions that omit the February 2013tests, using only data from the threeAugust tests for 2010 through2012.ThisaddressesconcernsthataFebruaryenddatemaygenerateadifferencein learning relative to testing students at grade level competencies at thebeginningofthenextschoolyear.Finally,thelasttwocolumnsreverttoourbasesampleofsmallschoolsandfourtestdates,butnowalsoincludetestresultsforstudentsingrade6. Theseregressionsrevealthattheresultsarerobusttothesedifferentsamples. The effect of exposure to Nali Kali on language test scores is

22  

statisticallysignificantandsimilar inmagnitudeinallspecifications.Thereisnostatisticallysignificanteffectonmathematicstestscores.5.3 EstimatesoftheeffectofNaliKalionnon‐cognitiveskills Table 11 reports results from our preferred regression for threenon‐cognitiveskills:communication,socialskillsandleadership.Theresultsreveal a statistically significant effect on leadership skills. Though thecoefficient on both communication and social skills is positive, it is notstatisticallysignificantatconventionallevels.Theseresultsdonotchangeifwe replace the cohort‐specific time trends with cohort‐year interactions(table12).Eveninthisverygeneralspecification,yearsofexposuretoNaliKalisignificantlyincreaseleadershipskills,buthavenosignificanteffectoncommunicationorsocialskills. Theseresultsareatoddswithanearlierdraftversionofthispaper,based on early rounds of the data (omitting the 2012‐13 rounds). Thisrestricted samplegeneratedweakpositiveeffectsofNaliKali on languageandmathematicstestscores,andstrongpositiveeffectsoncommunication,socialandleadershipskills.Toexplorereasonsforthisdivergence,weturnnowtoregressionresultsthatallowtheeffectofNaliKalitovarybygrade,across scheduled caste and tribe students and other students, and by thequartileofthetestscoredistributioninwhichthestudentwasplacedatthestartofthesurvey.5.4 Heterogeneityinregressionestimates5.4.1 VariationbygradeTable13allowsthecoefficientonyearsofexposuretoNaliKalitovarybygrade, and reports regression results from our preferred specification forlanguage andmathematics, aswell as for communication, social skills andleadership. F tests, reported at the bottom of the table, reject the nullhypothesisoftheequalityoftheeffectoftheprogramacrossgrades.

23  

In contrast to the previous tables that indicated no statisticallysignificant effects of the program on mathematics scores and oncommunication and social skills, this table reveals positive effects for alltests, butonly at lowergrades; estimated coefficientson the effect ofNaliKaliarenegativeforstudentsingrade5.Themaximumimprovementintestscoresforbothlanguageandmathematicsisreportedforstudentsingrade3.Thesameistrueforleadershipskills.Forcommunicationandsocialskills,scoresarehighestingrade2. These results thus suggest strong effects of the program in earlygrades(relativetostudentswho,atthesamegradelevel,hadlowerlevelsofexposure to the program because of its phasing in over time and acrossschools). These effects appear to be lost by grade 5, explaining theinsignificanteffectoftheprogramin(previous)tablesthataverageoverallgrades.Thisalsoexplainsthedifferenceintheresultsofthispaperandtheearlierdraftversion.Thelatteruseddatafromearlyroundsonly,withfewobservationsforstudentsingrade4andnoneforstudentsingrade5.

5.4.2 VariationbyabilitylevelWe next explore variation in results by initial ability level.We start withregressions that allow the coefficient on years of exposure to vary acrossstudents from scheduled castes and tribes, relative to other students,exploitingthestrongcorrelationbetweencasteandinitialabilitylevels.Theresults in table 14 suggest a stronger effect of the program on scheduledcasteandtribestudentsrelative tostudents fromhighercasteswho likelyenter schoolswith better academic and non‐academic skills. For studentsfromscheduledcastesandtribes,theprogramhasastatisticallysignificanteffect on language andmathematics test scores. It also has a positive andsignificant effect on language for students from other castes, though themagnitudeoftheeffectismarginallylower.Fortheotherskillsweconsider,the results similarly indicate a stronger effect on leadership skills forstudentsfromscheduledcastesandtribes.However,Ftestsreportedatthebottom of the table fail to reject the null hypothesis of equality of thecoefficientacrosscastes.

24  

To furtherexplorevariation in the effectivenessof theprogrambyabilitylevel,table15usestheAugust2010dataasabaseline,andclassifiesstudentsbytheirpositioninthedistributionoftestscoresfromthisround.The regression sample is reduced, sincebaselinedata is availableonly forthreecohortsandbecausewenowlimittheanalysistotestsreportedinthesubsequent rounds only (August 2011 and 2012 and February 2013).Despite this reduction in sample size, the same broad pattern emerges.Program effects are stronger for students at the bottom of the academicdistribution but much weaker for students at the top. For language, thecoefficientsarestatisticallysignificantonly forstudents in thebottomtwoquartiles, turning negative, but insignificantly so, for students in the topquartile.Ftests(reportedatthebottomofthetable)rejectthehypothesisofequalityofthecoefficientacrossabilityquartiles.Theeffectonmathematicstest scores is insignificant for all quartiles, though this may well be theconsequenceofthesignificantreductioninsamplesizeandtheremovalofdatafromonesurveyround.5.4.3 Grade‐levelcompetenciesrelativetolowergradesExploitingtheavailabilityofscores,foreachstudent,thattestcompetenciesatcurrentaswellaslowergrades,wearealsoabletoexplorethehypothesisthat theprogramenhances low‐level learning,butnotnecessarily learningof higher‐level material. Table 16 reports results from two regressions(separately for language and mathematics). The first regression, for eachsubject, reports students’ performance in questions that test learning onmaterialproscribedforthegradeinwhichthestudentiscurrentlyenrolled.Thesecondregressionreportsdeterminantsofachievement in thesubjectmaterialcoveredinthegradepriortothecurrentgrade. Though the coefficients are positive in all regressions, for bothlanguage andmathematics, the results are statistically significant only forlanguageinthegradelowertotheonethestudentiscurrentlyenrolledin.Though the results formathematics similarly suggest that the program is

25  

more successful in ensuring lower‐grade skills, the coefficient on years ofexposuretoNaliKaliisstatisticallysignificantonlyatthe13%level. The results confirm the hypothesis that the Nali Kali instructionalmethoddoesgenerategreater learningofbasicskills,butdoesnothelp inachievingtheskillsrequiredforhigherlevelwork.Itisworthnotingthatifwehadrestrictedouranalysistotestsofmaterialspecifictothegradethatthestudentwasenrolledin,asisthecommonpractice,wewouldnothavefoundanysignificanteffectsoftheprogramonlearning.5.5 ComparingestimatestosimpleevaluationsbasedoncohorteffectsAdistinctadvantageoftherichnessofthedatawecollectedforthisprojectis the ability to control for grade, cohort and year effects, as well as fordifferences between small and large schools, and yet identify the effect oftheprogram.Thisabilitysignificantlyaddstothecredibilityofourempiricalanalysis. It isworthasking,however, ifouranalysisgenerates results thatare significantly different from those thatwould obtain from simpler, lessdata‐intensivemethodsthatprimarilyexploitvariationintestscoresacrosscohortsthatdifferintheirexposuretoNaliKali. We present results from two simpler methodologies. The firstcompares students from cohorts 1 and 2, in the same survey year, whencohort1studentswereingrade5andcohort2studentsingrade4.Theonlystudentsincohort1whohadanyexposuretoNaliKaliwerethoseinsmallschools,whileallcohort2studentshadatleasttwoyearsofexposuretotheprogram. A comparison of outcomes across these two cohorts couldthereforebeconstruedasa“NaliKalieffect.”However,itconfoundsanyrealeffect of theprogramwith a cohort effect, and alsowith anydifference inlearning across grades. Despite this shortcoming, this methodology iscommonly followed in evaluation studies, since it requires just one cross‐section of data, and can be completed in theminimal amount of time. Insharp contrast to our results, the results from this regression, reported inthefirstsetofcolumnsintable17,revealverystrongeffectsoftheprogramonbothLanguageandMathematicstestscores.

26  

Asecondsetofresultsusesthesametwocohortsofstudentsbuttwoyears of data, basing evaluation on test scores of these two cohortswheneachwas in grade 5 and thereby eliminating grade‐specific effects.Whilethismethodologyrequiresalongertimeperiod(twoyearsinsteadofone),costsremainminimalsincetestingisonlydoneontwocohortsofstudents,witheachcohortbeing tested justonce.Again, theresults, reported in thelasttwocolumnsoftable17,arestrikinglydifferentfromourresults.Theysuggestaverystrongeffectonmathematicstestscores,butaninsignificanteffectonlanguageachievement. Theseresults,becausetheyareunabletodistinguishtheeffectoftheprogramfromcohortandyeareffects,are likely tobebiased. Specifically,theyascribethegenerallyhigherlevelsoflearningatlowergrades(grade4relativetograde5)bothinacross‐sectionandforanyspecificcohorttotheprogram,sinceyoungercohortshavegreaterexposuretoNaliKalithandooldercohorts.Withouttheuseofmultipleyearsofdataonseveraldifferentcohorts,itisimpossibletoseparatelyidentifyprogrameffects.

6. ConclusionThis paper presents results from one of the few empirical evaluations ofActivityBasedLearningonlanguageandmathematicstestscores,aswellason the development of non‐cognitive skills: communication, social andleadership skills. Our analysis exploits the phasing in of a program ofActivity Based Learning by the Karnataka Government in early primaryclasses.ThephasingingeneratesvariationinyearsofexposuretoNaliKaliacross cohorts in any given grade, but also within cohort variationgeneratedbytheearlyinitiationoftheprograminsmallschoolsinthestate. Wefindthattheprogramhadstrongsignificanteffectsonlanguagetest scores and leadership skills, but insignificant effects onmathematics,communicationandsocialskills.Exploringheterogeneityintheresults,wefind that there are positive effects of the program on mathematics and

27  

languages, as well as on all non‐cognitive skills, but that these areconcentratedinearlygrades;achievementsinearlygradesdonotpersistasstudent move into higher grades. Moreover, there is some evidence tosupportthehypothesisthatbenefitsaregreaterforstudentswhostartoffinthe lower quartiles of the distribution of language and mathematics testscores. Finally, our results suggest that the program helps students tomastercompetenciesexpectedatlowergrades,buthasnosignificanteffectin ensuring achievement at standards expected of the current grade inwhichthestudentisenrolled. Going forward, thesuccessof theprogramwouldrequireefforts toensurethatthebenefitsoftheprogramrecordedinearlygradespersist.Ouranalysisfurthersuggeststheneedtoimplementprocedurestoensurethatstudentsatthetopoftheabilitydistributionalsobenefitfromtheprogram.

28  

ReferencesAssessment Survey Evaluation Research Centre. 2011. Annual Status ofEducationReport2011.Duflo, Esther. 2001. “Schooling and Labor Market Consequences of SchoolConstruction in Indonesia: Evidence from an Unusual Policy Experiment.”AmericanEconomicReview91(4):795‐813.Macchiwalla, Tasqeen. (date unknown). “Nali‐Kali – A Not So SilentRevolutionforJoyfulLearning.”Availableat:http://planningcommission.gov.in/reports/sereport/ser/seeds/seed_edu.pdfSchoolScape,Centre forEducatorsandSarvaShikshaAbhiyan,Governmentof Tamil Nadu. 2009. “Activity Based Learning: Effectiveness of ABL underSSA.”Manuscript.

29  

 Table 1: Mean scores by cohort and grade 

Cohort / grade  Language  Mathematics  Communication Social skills  Leadelrship 

Cohort 1 (gr 3 in 2009‐10) 

         

Grade 3 (3224) 

39.59 (22.62) 

39.16 (16.35) 

47.75 (25.28) 

48.76 (26.76) 

24.53 (29.86) 

           Grade 4 (2757) 

38.85 (23.91) 

29.31 (16.42) 

63.25 (20.25) 

59.29 (22.78) 

30.65 (31.20) 

           Grade 5 (2628) 

36.34 (23.17) 

29.46 (17.03) 

54.83 (13.63) 

53.78 (18.20) 

46.21 (19.53) 

           Grade 6 (2421) 

32.90 (20.40) 

36.28 (20.15) 

57.39 (13.32) 

55.20 (17.10) 

48.90 (18.36) 

           Cohort 2 (gr 2 in 2009‐10) 

         

Grade 2 (2869) 

57.56 (19.17) 

54.63 (18.96) 

47.14 (24.89) 

49.44 (26.56) 

22.73 (28.59) 

           Grade 3 (2526) 

43.94 (21.48) 

48.93 (20.12) 

62.95 (20.36) 

59.27 (22.86) 

30.46 (30.99) 

           Grade 4 (2639) 

40.96 (21.47) 

33.67 (16.65) 

54.60 (14.25) 

54.40 (20.51) 

44.97 (22.17) 

           Grade 5 (2666) 

36.45 (21.54) 

40.78 (20.24) 

56.00 (12.35) 

54.01 (16.06) 

48.19 (16.89) 

           Cohort 3 (gr 1 in 2009‐10) 

         

Grade 1 (2527) 

70.96 (23.53) 

47.88 (20.95) 

43.08 924.59) 

45.37 (25.70) 

21.27 (27.90) 

           Grade 2 (2420) 

60.45 (18.24) 

63.19 (19.11) 

62.22 (19.55) 

58.60 (21.94) 

28.65 (29.75) 

           Grade 3 (2604) 

47.78 (22.33) 

49.42 (19.81) 

55.19 (14.59) 

53.91 (20.05) 

45.87 (21.21) 

           Grade 4 (2591) 

44.50 23.48) 

40.56 (19.94) 

54.74 (12.04) 

53.57 915.94) 

47.91 (17.00) 

           

30  

Table 1: Mean scores by cohort and grade (continued) 

Cohort / grade  Language  Mathematics  Communication Social skills  Leadership 

Cohort 4 (gr 1  in 2010‐11) 

         

Grade 1 (2409) 

71.23 (22.98) 

60.07 (19.08) 

58.08 (21.27) 

53.50 (24.15) 

25.47 (29.18) 

           Grade 2 2656)  

56.92 (19.22) 

62.65 (18.39) 

53.29 (13.50) 

53.06 (19.74) 

44.00 (21.46) 

           Grade 3 (2635) 

52.03 (22.96) 

50.75 (20.16) 

54.38 (13.11) 

52.95 (17.07) 

47.70 (18.11) 

           

 

Note: Standard deviations in parentheses. Figures in parentheses in column 1 are sample sizes. Scores for language and mathematics record percentage performance on questions that test grade level competencies as well as competencies for the immediately lower grade. For example, test scores for students in grade 4 report scores from questions testing both grade 4 and grade 3 material.  

31  

Table 2: Years of exposure to Nali Kali, by cohort and grade

Years of exposure to Nali Kali by grade and cohort (at end of year)

Cohort Grade in 2012-13

Small school

Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 1 6 No -- -- -- -- -- Yes 2 2 2 2 2 2 5 No 1 2 2 2 2 Yes 2 3 3 3 3 3 4 No 2 3 3 3 3 Yes 2 3 3 3 3 4 3 No 2 3 3 3 3 Yes 2 3 3 3 3

Table 3: Years of exposure to Nali Kali by grade and survey year

Years of exposure to Nali Kali by grade and year (at end of year)

Year Small school

Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 2009-10 No 1 (C2) 0 (C1) -- -- -- Yes 2 (C2) 2 (C1) -- -- -- 2010-11 No 2 (C3) 2 (C2) 0 (C1) -- -- Yes 2 (C3) 3 (C2) 2 (C1) -- -- 2011-12 No 2 (C4) 3 (C3) 2 (C2) 0 (C1) -- Yes 2 (C4) 3 (C3) 3 (C2) 2 (C1) -- 2012-13 No -- 2 (C4) 3 (C3) 2 (C2) 0 (C1) Yes 2 (C4) 3 (C3) 3 (C2) 2 (C1)

32  

 

 

Table 4: Continuation rates, by year, cohort and school size  

Year / school size  All cohorts  Cohort 1 (grade 6 in year 3) 

Cohort 2 (grade 5 in year 3) 

Cohort 3 (grade 4 in year 3) 

Year 2         All schools  0.89 

(0.32) 0.86 (0.34) 

0.88 (0.33) 

0.92 (0.27) 

Large schools  0.90 (0.30) 

0.88 (0.32) 

0.89 (0.31) 

0.93 (0.26) 

Small schools  0.88 (0.33) 

0.85 (0.36) 

0.87 (0.34) 

0.92 (0.27) 

         Year 2         All schools  0.80 

(0.40) 0.73 (0.45) 

0.82 (0.39) 

0.86 (0.35) 

Large schools  0.83 (0.38) 

0.81 (0.40) 

0.83 (0.38) 

0.85 (0.35) 

Small schools  0.77 (0.42) 

0.67 (0.47) 

0.81 (0.40) 

0.86 (0.34) 

         

 Note: small schools are those with fewer teachers than grades, and hence with multi‐grade instruction. 

33  

 

Table 5: Attendance rates by year, cohort and school size  

Cohort / school size  August 2010  August 2011  August 2012  February 2013 

All schools         Cohort 1 (grade 6 in 2012‐13) 

0.83 (0.37) 

0.71 (0.46) 

0.93 (0.26) 

0.95 (0.22) 

Cohort 2 (grade 5 in 2012‐13) 

0.83 (0.37) 

0.73 (0.45) 

0.93 (0.26) 

0.96 (0.19) 

Cohort 3 (grade 4 in 2012‐13) 

0.81 (0.39) 

0.75 (0.43) 

0.94 (0.24) 

0.95 (0.21) 

         Large schools         Cohort 1 (grade 6 in 2012‐13) 

0.84 (0.37) 

0.69 (0.46) 

0.91 (0.28) 

0.95 (0.21) 

Cohort 2 (grade 5 in 2012‐13) 

0.83 (0.37) 

0.73 (0.44) 

0.91 (0.29) 

0.97 (0.16) 

Cohort 3 (grade 4 in 2012‐13) 

0.78 (0.41) 

0.70 (0.46) 

0.93 (0.26) 

0.96 (0.20) 

         Small schools         Cohort 1 (grade 6 in 2012‐13) 

0.83 (0.38) 

0.72 (0.45) 

0.94 (0.24) 

0.95 (0.23) 

Cohort 2 (grade 5 in 2012‐13) 

0.83 (0.37) 

0.72 (0.45) 

0.94 (0.23) 

0.95 (0.21) 

Cohort 3 (grade 4 in 2012‐13) 

0.84 (0.37) 

0.78 (0.41) 

0.95 (0.22) 

0.95 (0.22) 

         

 Note: Small schools are those with fewer teachers than grades. Attendance rates are reported only amongst the sample of students still enrolled in schools and attending on a regular basis (i.e., excluding drop outs). Statistics are based on attendance in language tests. Attendance in mathematics tests are virtually identical to those reported for language. Standard deviations in parentheses. 

34  

 

Table 6: Sample participation, by cohorts and years of exposure to Nali Kali  

Years of exposure to Nali Kali Cohort 

0  1  2  3 

Cohort 1 (grade 6 in last survey year) 

0.72 (0.45) 

0.68 (0.47) 

0.77 (0.42) 

‐‐ 

         Cohort 2 (grade 5 in last survey year) 

‐‐  0.76 (0.43) 

0.77 (0.42) 

0.84 (0.37) 

         Cohort 3 (grade 4 in last survey year) 

‐‐  0.77 (0.42) 

0.80 (0.40) 

0.81 (0.39) 

         

 Note: Sample participation rates are calculated for those students in the survey in year 1, and reflect both drop out over the years as well as absenteeism for the test. Data reported are based on language tests; data from attendance in mathematics tests are virtually identical. Standard deviations in parentheses.

35  

 Table 7: Probit estimates of the effect of Nali Kali years on sample selection (Dependent Variable: Attendance at tests in the survey period)  

Variable  Regression 1  Regression 2  Regression 3  Regression 4  Regression 5 

Years Nali Kali  0.06 (0.05) 

0.06 (0.05) 

0.06 (0.05) 

0.05 (0.06) 

0.04 (0.06) 

Male  ‐0.06* 

(0.02) ‐0.03 (0.02) 

‐0.03 (0.08) 

‐0.03 (0.03) 

‐0.03 (0.03) 

Age  0.15* 

(0.01) 0.15* 

(0.01) 0.15* 

(0.01) 0.15* 

(0.01) 0.15* 

(0.01) SC/ST  ‐0.05 

(0.03) 0.01 (0.05) 

0.01 (0.05) 

0.01 (0.05) 

0.02 (0.05) 

Early adoption eligible school  

0.10 (0.08) 

0.04 (0.08) 

0.04 (0.08) 

0.04 (0.08) 

0.27* 

(0.14) Time trend  ‐‐  ‐‐  ‐0.30* 

(0.05) ‐‐  ‐‐ 

Cohort 1*time trend 

‐‐  ‐‐  ‐‐  ‐0.21 (0.26) 

‐0.20 (0.26) 

Cohort 2*time trend 

‐‐  ‐‐  ‐‐  ‐0.18 (0.13) 

‐0.16 (0.13) 

Cohort 3*time trend 

‐‐  ‐‐  ‐‐  ‐0.39* 

(0.06) ‐0.37* 

(0.06) Cohort 4*time trend 

‐‐  ‐‐  ‐‐  ‐0.44* 

(0.17) ‐0.44* 

(0.18) Eligible school*time trend 

‐‐  ‐‐  ‐‐  ‐‐  ‐0.10+ 

(0.06)            Cohort controls  Yes  Yes  Yes  Yes  Yes Grade controls  Yes  Yes  Yes  Yes  Yes Year controls  Yes  Yes  Yes  Yes  Yes District fixed effects 

yes  yes  yes  Yes  Yes 

Sample  Full  Small schools  Small schools  Small schools  Small schools Sample size  34,991  19,305  19,305  19,305  19,305 Wald χ2 (prob > χ2) 

932.67 (0.00) 

549.20 (0.00) 

549.20 (0.00) 

628.81 (0.00) 

622.04 (0.00) 

 Note: All regressions include size, size squared and proportion scheduled caste and tribe for the student’s cohort (in grade 1), for the school, and for primary grades (grades 1‐5). Other regressors are: the number of classes in the school, availability of toilets and drinking water, distance to the cluster  and  block office, proportion of teachers who are male and  from  scheduled  castes and  tribes, and  indicator variables  for whether  the  school  is a  lower primary school (grades 1‐5), or a higher primary school with grades up to 7 (higher primary schools with 8 grades are the omitted category), and for whether it is the  located in the main village or a hamlet. Small school sample includes only schools with less than 150. Students (LPS), less than 210 for HPS schools with 7 grades, or less than 240 students  if HPS with 8 grades. Regression sample for all regressions excludes students  in grade 1 and grade 6.Standard errors are clustered at the school level.   *Significant at 5% level    +Significant at 10% level 

36  

 

Table 8: Determinants of Language test scores  

Variable  Regression 1  Regression 2  Regression 3  Regression 4 

Years Nali Kali  0.12 (0.97) 

2.27* 

(0.99) 2.20* (0.00) 

2.27* (0.99) 

Male  ‐3.32* 

(0.56) ‐3.34* 

(0.56) ‐3.33* 

(0.56) ‐3.33 (0.56) 

Age  0.37 (0.30) 

0.38 (0.31) 

0.38 (0.31) 

0.40 (0.31) 

SC/ST  ‐3.62* 

(0.70) ‐3.65* 

(0.70) ‐3.64 (0.70) 

‐3.65 (0.70) 

Early adoption eligible school  

3.48+ 

(2.12) 2.96 (2.12) 

3.93 (2.90) 

2.97 (2.12) 

Time trend  3.23* 

(1.22) ‐‐  ‐‐  ‐‐ 

Cohort 1*time trend  ‐‐  ‐3.60* 

(1.63) ‐3.61* 

(1.63) ‐‐ 

Cohort 2*time trend  ‐‐  ‐6.77* 

(1.21) ‐6.75* 

(1.21) ‐‐ 

Cohort 3*time trend  ‐‐  ‐8.37* 

(1.44) ‐8.30* 

(1.45) ‐‐ 

Cohort 4*time trend  ‐‐  1.78 

(1.30) 1.81 (1.30) 

‐‐ 

Eligible school*time trend 

‐‐  ‐‐  ‐0.42 (1.11) 

‐‐ 

Cohort –year interactions  ‐‐  ‐‐  ‐‐  Yes Cohort controls  Yes  Yes  Yes  ‐‐ Grade controls  Yes  Yes  Yes  ‐‐ Year controls  Yes  Yes  Yes  ‐‐ District fixed effects  Yes  Yes  Yes  Yes Sample size  15,218  15,218  15,218  15,211 Regression F (Prob > F) 

41.16 (0.00) 

39.58 (0.00) 

39.10 (0.00) 

39.26 (0.00) 

F tests for: joint significance of cohort time trends / cohort‐year interactions 

‐‐  62.86 (0.00) 

62.21 (0.00) 

65.73 (0.00) 

equality of cohort‐time trend interactions 

‐‐  80.43 (0.00) 

79.37  (0.00) 

‐‐ 

 Note: For sample and additional regressors, please see note to table 7. Standard errors (in parentheses) are clustered at the school level.    *Significant at 5% level    +Significant at 10% level 

37  

 

Table  9: Determinants of Mathematics test scores  

Variable  Regression 1  Regression 2  Regression 3  Regression 4 

Years Nali Kali  1.69* 

(0.89) 1.25 (0.91) 

1.29 (0.93) 

1.25 (0.91) 

Male  ‐1.08* 

(0.41) ‐1.08* 

(0.41) ‐1.08* 

(0.41) ‐1.08* 

(0.41) Age  0.32 

(0.24) 0.33 (0.24) 

0.33 (0.24) 

0.35 (0.24) 

SC/ST  ‐2.76* 

(0.61) ‐2.76* 

(0.61) ‐2.76* 

(0.61) ‐2.76 (0.61) 

Early adoption eligible school  

1.05 (1.84) 

1.18 (1.86) 

0.48 (3.20) 

1.18 (1.86) 

Time trend  ‐6.44* 

(1.22) ‐‐  ‐‐  ‐‐ 

Cohort 1*time trend  ‐‐  ‐14.60* 

(1.70) ‐14.59* 

(1.70) ‐‐ 

Cohort 2*time trend  ‐‐  ‐9.48* 

(1.26) ‐9.49* 

(1.25) ‐‐ 

Cohort 3*time trend  ‐‐  ‐6.70* 

(1.61) ‐6.74* 

(1.61) ‐‐ 

Cohort 4*time trend  ‐‐  ‐4.21* 

(1.40) ‐4.23* 

(1.39) ‐‐ 

Eligible school*time trend 

  ‐‐  0.29 (1.22) 

‐‐ 

Cohort‐year interactions 

‐‐  ‐‐  ‐‐  Yes 

Cohort controls  Yes  Yes  Yes  ‐‐ Grade controls  Yes  Yes  Yes  ‐‐ Year controls  Yes  Yes  Yes  ‐‐ District fixed effects  Yes  Yes  Yes  yes Sample size  15,309  15,309  15,309  15,309 Regression F (Prob > F) 

68.22 (0.00) 

69.54 (0.00) 

67.97 (0.00) 

69.31 (0.00) 

F tests for: joint significance of cohort time trends / cohort‐year interactions 

‐‐  85.23 (0.00) 

84.46 (0.00) 

93.22 (0.00) 

equality of cohort –time trend interactions 

‐‐  105.86 (0.00) 

105.55 (0.00) 

‐‐ 

 Note: For sample and additional regressors, please see note to table7. Standard errors (in parentheses) are clustered at the school level.   *Significant at 5% level    +Significant at 10% level  

38  

 

           Table 10: Sensitivity Analysis  

Sample: all schools  Sample: August tests only (3 years) 

Sample: Base + 6th grade scores 

Variable 

Language  Mathematics  Language  Mathematics  Language  Mathematics

Years Nali Kali  1.98+ 

(1.13) 1.18 (1.13) 

2.01* 

(1.05) 0.86 (0.96) 

2.10* (1.05) 

0.57 (0.93) 

Male  ‐2.89* 

(0.48) ‐0.73* 

(0.37) ‐3.05* 

(0.57) ‐0.81* 

(0.42) ‐3.39* 

(0.55) ‐1.28* 

(0.39) Age  ‐0.007 

(0.23) 0.01 (0.20) 

0.20 (0.33) 

0.40+ 

(0.23) 0.41 (0.30) 

0.40+ 

(0.24) SC/ST  ‐2.66* 

(0.54) ‐1.95* 

(0.44) ‐4.00* 

(0.80) ‐3.11* 

(0.70) ‐3.71 (0.73) 

‐2.86* 

(0.61) Early adoption eligible school  

4.04 (2.92) 

3.22 (3.51) 

10.93* 

(3.55) 5.45+ 

(2.99) 3.64 (2.92) 

0.72 (3.15) 

Cohort 1*time trend 

‐4.28* 

(1.42) ‐12.90* 

(1.78) ‐7.73* 

(0.88) ‐10.45* 

(0.79) ‐3.67 (1.35) 

‐8.02* 

(1.40) Cohort 2*time trend 

‐7.13* 

(1.02) ‐7.51* 

(1.28) ‐5.40* 

(1.13) ‐3.71* 

(1.30) ‐6.78 (1.29) 

‐3.98* 

(1.30) Cohort 3*time trend 

‐7.18* 

(1.28) ‐3.85* 

(1.46) ‐2.29+ 

(1.31) ‐1.36 (1.18) 

‐8.30 (1.54) 

‐2.33 (1.77) 

Cohort 4*time trend 

1.38 (1.07) 

‐2.67+ 

(1.44) ‐2.12+ 

(1.16) 0.28 (1.00) 

1.75 (1.23) 

‐1.58 (1.30) 

Eligible school*time trend 

‐0.72 (1.01) 

‐0.31 (1.33) 

‐3.62* 

(1.48) ‐1.78 (1.22) 

‐0.33 (1.08) 

0.44 (1.10) 

             Cohort controls  Yes  Yes  Yes  Yes  Yes  Yes Grade controls  Yes  Yes  Yes  Yes  Yes  Yes Year controls  Yes  Yes  Yes  Yes  Yes  Yes District fixed effects 

yes  Yes  Yes  yes  Yes  Yes 

Sample size  27,423  27,586  12,070  12,161  16,133  16,377 Regression F (Prob > F) 

55.52 (0.00) 

75.56 (0.00) 

46.73 (0.00) 

81.32 (0.00) 

99.92 (0.00) 

72.43 (0.00) 

 Note: For additional regressors, please see note to table 7. Standard errors (in parentheses) are clustered at the school level.   *Significant at 5% level    +Significant at 10% level  

39  

 

      Table  11: Determinants of non‐cognitive skills with  time‐trend  

Variable  Communication  Social skills  Leadership 

Years Nali Kali  0.82 (1.41) 

1.07 (1.57) 

3.62* 

(1.73) Male  0.06 

(0.34) ‐0.54 (0.38) 

‐0.73+ 

(0.39) Age  0.31 

(0.23) 0.56* 

(0.27) 0.54+ 

(0.31) SC/ST  ‐1.41* 

(0.55) ‐1.59* 

(0.53) ‐2.22* 

(0.57) Early adoption eligible school   6.40* 

(3.26) 3.06 (3.76) 

‐7.89* 

(3.89) Cohort 1*time trend  0.19 

(2.09) ‐1.56 (2.53) 

2.34 (2.59) 

Cohort 2*time trend  ‐0.37 (1.41) 

‐3.00+ 

(1.72) ‐0.11 (1.90) 

Cohort 3*time trend  ‐0.82 (1.73) 

‐4.18+ 

(2.23) ‐3.57+ 

(2.01) Cohort 4*time trend  ‐0.96 

(1.57) ‐2.29 (1.95) 

‐0.29 (2.00) 

Eligible school*time trend  ‐1.89+ 

(1.13) ‐0.81 (1.26) 

2.22 (1.53) 

       Cohort controls  Yes  Yes  Yes Grade controls  Yes  Yes  Yes Year controls  Yes  Yes  Yes District fixed effects  Yes  Yes  Yes Sample size  14,513  14,490  14,396 Regression F (Prob > F) 

13.30 (0.00) 

10.87 (0.00) 

26.40 (0.00) 

F tests for: joint significance of cohort time trends 

 0.73 (0.57) 

1.29 (0.29) 

1.87 (0.12) 

Equality of cohort‐time trend interactions 

0.85 (0.47) 

0.42 (0.74) 

2.15 (0.09) 

 Note: Sample is students who took the test. For additional regressors, please see note to table 7. Standard errors (in parentheses) are clustered at the school level.    *Significant at 5% level    +Significant at 10% level 

40  

 

Table 12 : Determinants of non‐cognitive skills with  cohort‐year interactions  

Variable  Communication  Social skills  Leadership 

Years Nali Kali  1.15 (1.34) 

1.37 (1.54) 

3.27+ 

(1.75) Male  0.06 

(0.35) ‐0.51 (0.38) 

‐0.70+ 

(0.39) Age  0.31 

(0.23) 0.57* 

(0.27) 0.54+ 

(0.31) SC/ST  ‐1.45* 

(0.55) ‐1.59* 

(0.53) ‐2.20* 

(0.57) Early adoption eligible school   2.13 

(1.50) 1.02 (1.92) 

‐2.89 (1.88) 

       Cohort‐year interactions  yes  Yes  Yes District fixed effects  Yes  Yes  Yes Sample  Full  Small schools  Small schools Sample size  14,545  14,522  14,428 Regression F (Prob > F) 

13.46 (0.00) 

10.80 (0.00) 

26.08 (0.00) 

F tests for: joint significance of cohort‐year interactions 

1.63 (0.09) 

1.00 (0.45) 

8.23 (0.00) 

 Note: Sample is students who took the test. For additional regressors, please see note to table 7. Standard errors (in parentheses) are clustered at the school level.    *Significant at 5% level    +Significant at 10% level 

41  

 Table  13: Heterogeneous effects of exposure to Nali Kali by grade  

Variable  Language  Mathematics  Communication Social skills  Leadership 

Years Nali Kali* grade 2  2.44 (2.91) 

0.72 (2.46) 

6.80* 

(3.32) 9.19* 

(3.49) 5.35 (4.75) 

Years Nali Kali*grade 3  4.32* 

(1.38) 2.62+ 

(1.48) 3.48* 

(1.83) 2.82 (2.03) 

8.88* 

(2.23) Years Nali Kali * grade 4 

2.71* 

(1.22) 1.85* 

(0.95) ‐0.71 (1.64) 

‐0.88 (1.81) 

2.95 (1.94) 

Year Nali Kali*grade 5  ‐1.92 (1.59) 

‐1.41 (1.76) 

‐4.23* 

(1.45) ‐1.98 (1.32) 

‐7.03* 

(1.77) Male  ‐3.33* 

(0.56) ‐1.07* 

(0.41) 0.06 (0.34) 

‐0.55 (0.38) 

‐0.72+ 

(0.39) Age  0.39 

(0.31) 0.33 (0.24) 

0.31 (0.23) 

0.56 (0.27) 

0.55+ 

(0.30) SC/ST  ‐3.65* 

(0.71) ‐2.76* 

(0.62) ‐1.43* 

(0.55) ‐1.60 (0.53) 

‐2.25* 

(0.56) Early adoption eligible school  

2.37 (3.11) 

‐0.37 (3.31) 

3.84 (3.39) 

0.72 (3.70) 

‐11.48* 

(4.05) Cohort 1*time trend  ‐1.32 

(4.09) ‐14.01* 

(3.69) 10.07* 

(4.63) 10.12* 

(5.05) 9.42 (6.61) 

Cohort 2*time trend  ‐0.88 (3.36) 

‐6.25+ 

(3.46) 11.33* 

(3.70) 7.63* 

(3.86) 15.93* 

(5.22) Cohort 3*time trend  ‐4.90+ 

(2.88) ‐5.12+ 

(2.75) 8.26* 

(3.30) 4.88 (3.51) 

6.93 (4.76) 

Cohort 4*time trend  3.55 (2.30) 

‐3.53+ 

(2.18) 4.67+ 

(2.60) 3.85 (2.92) 

4.99 (3.68) 

Eligible school*time trend 

0.22 (1.19) 

0.64 (1.27) 

‐0.90 (1.22) 

0.09 (1.26) 

3.62* 

(1.60) Cohort controls  Yes  Yes  Yes  Yes  Yes Grade controls  Yes  Yes  Yes  Yes  Yes Year controls  Yes  Yes  Yes  Yes  Yes District fixed effects  Yes  Yes  Yes  Yes  Yes Sample size  15,218  15,309  14,513  14,490  14,396 Regression F (Prob > F) 

39.91 (0.00) 

67.49 (0.00) 

14.67 (0.00) 

10.85 (0.00) 

28.29 (0.00) 

F tests for: joint significance of years NK and grade interactions 

3.26 (0.01) 

1.43 (0.23) 

4.54 (0.002) 

2.88 (0.02) 

15.59 (0.00) 

equality of grade interactions 

3.31 (0.02) 

1.19 (0.31) 

5.63 (0.001) 

3.77 (0.01) 

20.09 (0.00) 

 Note: For sample and additional regressors, please see note to table 7 Standard errors (in parentheses) are clustered at the school level.   *Significant at 5% level    +Significant at 10% level  

42  

 

Table 14 : Heterogeneous effects of exposure to Nali Kali by caste  

Variable  Language  Mathematics  Communication  Social skills  Leadership 

Years Nali Kali* scheduled caste, tribe 

2.69* 

(1.11) 1.76+ 

(1.05) 1.50 (1.45) 

1.81 (1.64) 

3.96* 

(1.72) 

Years Nali Kali*other castes 

2.02* 

(1.04) 1.12 (0.92) 

0.58 (1.42) 

0.80 (1.58) 

3.50* 

(1.76) Male  ‐3.34* 

(0.56) ‐1.08* 

(0.41) 0.06 (0.34) 

‐0.54 (0.38) 

‐0.74+ 

(0.39) Age  0.39 

(0.31) 0.33 (0.24) 

0.31 (0.23) 

0.57* 

(0.27) 0.54+ 

(0.31) SC/ST  ‐4.71* 

(1.26) ‐3.76* 

(1.02) ‐2.84* 

(1.14) ‐3.17 (1.17) 

‐2.95+ 

(1.03) Early adoption eligible school  

3.88 (2.90) 

0.44 (3.19) 

6.33* 

(3.23) 2.98 (3.73) 

‐7.93* 

(3.89) Cohort 1*time trend  ‐3.61* 

(1.64) ‐14.59* 

(1.71) 0.18 (2.09) 

‐1.58 (2.53) 

2.33 (2.59) 

Cohort 2*time trend  ‐6.76* 

(1.21) ‐9.49* 

(1.25) ‐0.37 (1.41) 

‐3.00+ 

(1.72) ‐0.12 (1.90) 

Cohort 3*time trend  ‐8.30* 

(1.45) ‐6.74* 

(1.61) ‐0.82 (1.74) 

‐4.17+ 

(2.24) ‐3.57+ 

(2.01) Cohort 4*time trend  1.81 

(1.31) ‐4.22* 

(1.40) ‐0.97 (1.57) 

‐2.29 (1.96) 

‐0.29 

(2.00) Eligible school*time trend 

‐0.41 (1.11) 

0.30 (1.22) 

‐1.87+ 

(1.12) ‐0.79 (1.25) 

2.23 (1.52) 

           Cohort controls  Yes  Yes  Yes  Yes  Yes Grade controls  Yes  Yes  Yes  Yes  Yes Year controls  Yes  Yes  Yes  Yes  Yes District fixed effects  Yes  Yes  Yes  Yes  Yes Sample size  15,218  15,309  14,513  14,490  14,396 Regression F (Prob > F) 

38.94 (0.00) 

66.70 (0.00) 

13.40 (0.00) 

11.02 (0.00) 

26.41 (0.00) 

F tests for equality of caste effects 

1.18 (0.28) 

1.37 (0.24) 

2.73 (0.10) 

2.53 (0.11) 

0.68 (0.41) 

 Note: Sample is students who took the test. For additional regressors, please see note to table 7. Standard errors (in parentheses) are clustered at the school level.    *Significant at 5% level    +Significant at 10% level 

43  

 

Table  15: Heterogeneous effects of exposure to Nali Kali by initial ability quartile  

Variable  Language  Mathematics 

Years Nali Kali* lowest quartile  2.46+ 

(1.50) 0.87 (1.16) 

Years Nali Kali * second quartile  3.15* 

(1.42) 1.28 (1.05) 

Years Nali Kali * third quartile  0.67 (1.43) 

0.57 (1.06) 

Years Nali Kali * top quartile  ‐0.26 (1.55) 

1.39 (1.06) 

Quartile 1  ‐27.85* 

(2.07) ‐11.03* 

(1.60) Quartile 2  ‐22.54* 

(1.62) ‐6.93* 

(1.33) Quartile 3  ‐11.00* 

(1.64) ‐2.70* 

(1.27) Male  ‐3.73* 

(0.54) ‐0.84* 

(0.44) Age  ‐0.01 

(0.46) 0.27 (0.33) 

SC/ST  ‐2.47 (0.63) 

‐1.96* 

(0.75) Early adoption eligible school   ‐6.21+ 

(3.75) ‐0.10 (5.04) 

Eligible school*time trend  2.18 (1.38) 

0.33 (1.81) 

Cohort controls  Yes  Yes Grade controls  Yes  Yes Year controls  Yes  Yes District fixed effects  Yes  Yes Sample size  10,369  10,557 Regression F (Prob > F) 

79.23 (0.00) 

60.21 (0.00) 

F test for joint significance of NK‐quartile interactions 

6.62 (0.00) 

1.18 (0.32) 

Equality of ability interactions  8.29 (0.00) 

1.07 (0.32) 

 Note: August 2010 test scores used as initial learning levels. Regressions are based on subsequent rounds of data only, excluding this year.  For sample and additional regressors, please see note to table 7.  All regressions also include interactions of indiators for cohort with a time trend. Standard errors (in parentheses) are clustered at the school level.   *Significant at 5% level   +Significant at 10% level 

44  

 

 Table 16 : Determinants of learning at grade level and at level for lower grade  

Language  Mathematics Variable 

Grade level  Lower grade  Grade level  Lower level 

Years Nali Kali  1.68 (1.14) 

2.71* 

(1.24) 0.99 (0.96) 

1.60 (1.07) 

Male  ‐3.28* 

(0.60) ‐3.40* 

(0.58) ‐1.12* 

(0.43) ‐1.04* 

(0.45) Age  0.57 

(0.39) 0.20 (0.27) 

0.61* 

(0.27) 0.05 (0.26) 

SC/ST  ‐4.09* 

(0.76) ‐3.20* 

(0.72) ‐2.73* 

(0.59) ‐2.79* 

(0.71) Early adoption eligible school  

3.49 (3.46) 

4.36 (2.77) 

‐0.64 (3.46) 

1.61 (3.36) 

Cohort 1*time trend  ‐9.52* 

(2.25) 2.30 (1.65) 

‐6.04* 

(2.14) ‐23.14* 

(1.65) Cohort 2*time trend  ‐8.49* 

(1.53) ‐5.02* 

(1.18) ‐6.41* 

(1.42) ‐12.57* 

(1.31) Cohort 3*time trend  ‐0.93 

(1.69) ‐15.67* 

(1.79) 1.23 (1.71) 

‐14.72* 

(1.82) Cohort 4*time trend  ‐5.84* 

(1.78) 9.47* 

(1.24) 0.33 (1.71) 

‐8.79* 

(1.33) Eligible school*time trend  ‐0.02 

(1.29) ‐0.81 (1.10) 

0.48 (1.37) 

0.10 (1.22) 

         Cohort controls  Yes  Yes  Yes  Yes Grade controls  Yes  Yes  Yes  Yes Year controls  Yes  Yes  Yes  Yes District fixed effects  Yes  Yes  Yes  Yes Sample size  15,216  15,216  15,309  15.309 Regression F (Prob > F) 

17.98 (0.00) 

96.36 (0.00) 

68.90 (0.00) 

63.06 (0.00) 

F tests for: joint significance of cohort time trends  

12.30 (0.00) 

144.92 (0.00) 

35.62 (0.00) 

130.83 (0.00) 

equality of cohort time trend interactions 

12.76  (0.00) 

192.37 (0.00) 

47.43 (0.00) 

164.39 (0.00) 

 Note: For sample and additional regressors, please see note to table7. Standard errors (in parentheses) are clustered at the school level.    *Significant at 5% level    +Significant at 10% level 

45  

 

Table 17:  Simple evaluation of Nali Kali based on cohort effects  

Sample: same year, different grades  Sample: grade 5 test scores Variable 

Language  Mathematics  Language  mathematics 

Cohort 2 (grade 2 in 2009‐10) 

6.86* 

(1.36) 6.07* 

(0.97) 2.56 (1.64) 

16.46* 

(1.95) 

         Male  ‐5.03* 

(0.87) ‐0.31 (0.62) 

‐3.75 (0.81) 

0.63 (0.67) 

Age  ‐0.33 (0.34) 

‐0.20 (0.25) 

‐0.26 (0.23) 

0.07 (0.23) 

SC/ST  ‐4.17* 

(0.91) ‐1.74* 

(0.61) ‐3.44* 

(0.82) 0.07 (0.23) 

         School size  0.20* 

(0.09) 0.04 (0.10) 

0.05 (0.07) 

‐0.08 (0.10) 

School size squared  ‐0.0004* 

(0.0002) ‐0.0001 (0.0002) 

‐0.0001 (0.0001) 

0.0001 (0.0002) 

School prop SC/ST  ‐54.59* 

(25.62) 0;33 

(25.37) ‐40.63* 

(18.63) ‐30.48 (20.29) 

Primary grades size  ‐0.37* 

(0.15) ‐0.06 (0.14) 

‐0.12 (0.11) 

0.11 (0.14) 

Primary grades size squared 

0.001* (0.0004) 

0.0002 (00.0004) 

0.0005+ 

(0.0003) ‐0.0001 (0.0004) 

Primary grades prop SC/ST 

48.53* 

(24.83) ‐4.89 (24.40) 

34.33+ 

(19.15) 19.10 (20.52) 

         District fixed effects  Yes  Yes  Yes  Yes Sample size  4986  4980  4998  4998 Regression F (Prob > F) 

18.89 (0.00) 

5.93 (0.00) 

11.52 (0.00) 

6.28 (0.00) 

 Note: Columns 2 and 3 report results for cohorts 1 and 2, for February 2012 tests, when cohort 1 was in grade 5 and cohort 2 in grade 4. Columns 3 and 4 report results from tests of these same cohorts, when each of them were in grade 5 (February 2012 for cohort 1 and February 2013 for cohort 2). Additional regressors are listed in the note to table 7. Standard errors (in parentheses) are clustered at the school level.    *Significant at 5% level    +Significant at 10% level 

46  

Figure 1: Language scores over years by initial ability  quartile and cohort   

02

04

06

08

01

00g

rad

e le

vel l

ang

scor

e

2010 2011 2012 2013year

Grade level Language scores, cohort 1

                         0

20

40

60

80

100

gra

de

leve

l lan

g sc

ore

2010 2011 2012 2013year

Grade level Language scores, cohort 2

 

 

02

04

06

08

01

00g

rad

e le

vel l

ang

scor

e

2010 2011 2012 2013year

Grade level Language scores, cohort 3

                     

02

04

06

08

01

00g

rad

e le

vel l

ang

scor

e

2011 2011.5 2012 2012.5 2013year

Grade level Language scores, cohort 4

 

Note: In initial year, cohort 3 was in grade 3, cohort 2 in  grade 2, and cohort 3 in grade 1. Cohort 4 was in grade 1 

in 2011.  

 

 

47  

Figure 2: Grade level math scores over years by initial ability quartile and cohort  

02

04

06

08

01

00g

rad

e le

vel m

ath

sco

re

2010 2011 2012 2013year

Grade Level Mathematics Scores, Cohort 1

              0

20

40

60

80

100

gra

de

leve

l ma

th s

core

2010 2011 2012 2013year

Grade Level Mathematics Scores, Cohort 2

 

 

 

02

04

06

08

01

00g

rad

e le

vel m

ath

sco

re

2010 2011 2012 2013year

Grade Level Mathematics Scores, Cohort 3

                

02

04

06

08

01

00g

rad

e le

vel m

ath

sco

re

2011 2012 2013year

Grade Level Mathematics Scores, Cohort 4

 

 

Note: In initial year, cohort 3 was in grade 3, cohort 2 in  grade 2, and cohort 3 in grade 1. Cohort 4 was in grade 1 

in 2011.  

48  

 

Figure 3: Grade‐specific test scores over years, by initial ability quartile and cohort, language and mathematics 

  

02

04

06

08

01

00g

rad

e 4

lang

sco

re

2010 2011 2012 2013year

Grade 4 Language scores, cohort 1

                

02

04

06

08

01

00g

rad

e 3

lang

sco

re

2010 2011 2012 2013year

Grade 3 Language scores, cohort 2

 

 

02

04

06

08

01

00g

rad

e 4

mat

h sc

ore

2010 2011 2012 2013year

Grade 4 Mathematics Scores, Cohort 1

                      

02

04

06

08

01

00g

rad

e 3

mat

h sc

ore

2010 2011 2012 2013year

Grade 3 Mathematics Scores, Cohort 2

 

 

Note: For each cohort, graphs show learning over time at the same grade level. For example. Graphs for cohort 1 

show results from test scores that test grade 4 competencies, over different years (grades). Similarly, graphs for 

cohort 2 show results only from testing at the grade 3 level, in different years 

49  

 

Figure 3: page from school report book charting daily progress of each student in language, mathematics and science.