Larry CondelliStephanie Cronen

American Institutes for Research, USA

LESLLA Sixth Annual Symposium

Cologne, GermanyAugust 26, 2010

1

Overview of PresentationSummary of Study Purpose and

Design Impact of the InterventionContent of InstructionStudent AttendanceDiscussion and Implications of

Findings

2

Evaluation of an ESL Literacy Intervention: Sam and PatStructured language approach Adapted from Wilson Reading

SystemNever before evaluated for ESL Literacy activities organized

around basal reader/workbookHeavily phonics-based

3

Sam and Pat: Instructional Approach

Direct instruction, transparentRules explained, modeled, practiced

Controlled text, vocabulary and grammarWords match phonics already learned

Sequential Easy to hard in defined steps

ESL instruction to support literacy

4

Sam and Pat: Literacy and Language Skills CoveredPhonics for reading and writingSight wordsOral reading for accuracy and fluencyReading comprehensionVocabularySpeaking and listeningGrammar

5

Research QuestionsHow effective is instruction using the

intervention in improving the English reading, speaking and listening skills of low-literate adult ESL learners?

Is the intervention more effective for certain groups of students (e.g., language, literacy level)?

Do differences in level of implementation of Sam and Pat and other instruction relate to variation in impacts?

6

Study Design10 adult ESOL centers across USAPaired intervention and “normal” ESL

literacy classes (34 total)Random assignment of students and

teachers1,344 students participated for one term

Minimum 5 hours/week 10-12 weeks instruction with approach

Other instruction also providedEach class conducted twice over a year

7

Student Flow in the Study

1. Student applies to center2. Assessed for NLL3. Recruited into study4. Gives informed consent 5. Random assignment to class 6. Pretests administered7. Instruction8. Posttests administered

8

IntakeNL Literacy

Post-test

Random Assignment

Recruited into StudyInformed consent

Pre-test

Instruction

Students AssessmentsPhonics and decoding

Word attackLetter/Word ID

Reading comprehensionVocabulary (ROWPVT)Listening, oral expression

9

Sam And Pat Teachers in the StudyAll teachers randomly assigned3-day teacher training on curriculumFollow-up visits by trainersClassroom observations

to monitor fidelityRefresher webinar at

start of second term

10

Students in the Study:“True” Literacy (LESLLA) Little or no literacy in native

languageLimited oral EnglishEducation: 0-3 yearsLanguages:

Haitian-CreoleSpanishBurmese Others

11

Students in the Study: Non-Roman Alphabet LiterateSome Literacy in native language with

non-Roman scriptMean education: 6 and more yearsLimited oral English Languages:

ArmenianArabicFarsiChinese Others

12

Student Test Results

13

Main Impact Analyses

Outcome Treatment Control Difference Effect Size P-value

WJID 440.81 442.09 -1.28 -0.025 0.573

WJWA 466.70 465.76 0.94 0.024 0.595

SARA Dec 13.25 13.37 -0.12 -0.011 0.809

WJPC 432.83 433.52 -0.70 -0.038 0.295

OWLS 17.80 17.79 0.02 0.002 0.974

ROWPVT 28.53 29.55 -1.02 -0.061 0.111

WJPVT 431.65 431.24 0.41 0.020 0.663

Sample size: 1137 students14

No significant impacts on literacy or language outcomes for full sample (below) and subsamples

But all students gained on pre-post tests

Pre-post Differences, All StudentsOutcome Mean

PretestMean Posttest

Difference P-value

WJID 428.315 442.122 13.808 0.000*WJWA 457.400 466.503 9.103 0.000*WJPC 427.061 433.780 6.719 0.000*OWLS 14.239 18.075 3.836 0.000*ROWPVT 22.898 29.285 6.387 0.000*N=1,113Sam & Pat=567Control=546

15

Lack of Impact of Sam and Pat

The study took place in a challenging environment that makes having/finding an impact difficult, but that represents realityLow exposure and instructional time in adult

ESL class Short class duration Limited instructional time and irregular attendance

Training teachers Lack of specialized training in literacy Short training time available

16

Instruction: What is taught?

17

Classroom ObservationsLiteracy

development instruction:Pre-literacyPhonicsFluencyReading

strategies & comprehension

Writing

ESL Instruction:Oral language

developmentGrammar, etc.VocabularySocio-cultural

knowledgeFunctional

literacy18

Instructional Content by Interval (%)Content Sam and Pat

MeanControl Mean Difference P-value

Reading Development

65.5 19.3 46.3 0.00

ESL Acquisition

27.3 67.6 -40.2 0.00

Other Content 5.5 28.8 -23.3 0.00

Link to Outside

1.3 7.4 -6.1 0.071

Use of NL 20.7 43.9 -23.2 0.00

Sample SizeIntervalsObservations

98031

103433

19

Instruction in Literacy ClassesIn control (“normal”) classes, little

literacy taught- Why?Importance of oral language for daily lifeStudent expressed needs Lack of teaching on how to teach LESLLA students

More diversity of instruction in control classesMore NL use and bringing in outsideLess constrained by curriculum and study

needs

20

Attendance

21

Attendance and OutcomesOutcome Regression

CoefficientP=value

WJID 0.104 0.00WJWA 0.071 0.00WJPC 0.043 0.00OWLS 0.027 0.00ROWPVT 0.028 0.00WJPVT 0.056 0.00Mean attendance hours (p= n.s.)

Sam & Pat: 79.4 Control: 71.9

N=1,137 ,Sam & Pat=587, Control=557

22

Effect of Attendance on Test ScoresInstruction and outcomes correlatedInstruction appears to have an effect on

learningRelationship is weakLarger effect on reading outcomes

May mean literacy gains more sensitive to gain, may be testing artifact

23

Implications of Possible Findings for Practice

and Research for LESLLA Students

24

Implications for LESLLA Practice Sam & Pat is no more effective than other

types of ESL literacy instructionUnder what conditions, then, would Sam & Pat be a

good choice for instruction? Attendance relates to instruction, weakly

Is this what you expect? How could it be increased?Teachers of literacy classes teach little literacy

How can we improve this --through curriculum and teacher professional development?

What are some potential explanations for these results from your perspective?

25

Thank you!Contacts

lcondelli@air.orgscronen@air.org

Enjoy Koeln and LESLLA

26