Development and Evaluation of Tier 2 Instruction to ...€¦ · Elizabeth Spencer Howard Goldstein...

Development and Evaluation of Tier Development and Evaluation of Tier 2 Instruction to Prevent Language 2 Instruction to Prevent Language and Literacy Delaysand Literacy DelaysElizabeth Spencer

Howard Goldstein

Sean Noe

Trina Spencer

The Ohio State University

Cooperative Agreement funded by Institute of Education Sciences

Presented at RTI Summit, 2011

Tier 2 Embedded Storybook Tier 2 Embedded Storybook InterventionsInterventions� Conceptualizing an RTI model in early childhood settings

� High-quality curricula needed to improve early language and literacy skills for early language and literacy skills for preschool children who are falling behind

� Development work

◦ Feasibility for implementation in classrooms

◦ Results of early efficacy studies

Challenges to EC RTIChallenges to EC RTI

� Pre-K settings vary (e.g., Head Start, public school programs, privately funded child care)◦ Who should implement measures and higher tier interventions?

� Personnel often lack training and expertise, are underpaid, and have high turnover ratesEvidence-based practices are not common-place� Evidence-based practices are not common-place

� Including teacher-directed instruction in Pre-K is often controversial

� Designing interventions that balance “developmentally appropriate practice” and instructional practices that are sufficient to boost children who might be struggling to acquire early language and literacy skills

To effectively implement response to intervention in early childhood…

…how should we design Tier 2 interventions?…how should we design Tier 2 interventions?

We need interventions that work in classrooms…We need interventions that work in classrooms…

…and that don’t place additional demands on teachers.…and that don’t place additional demands on teachers.

Children learn best when we teach explicitly…

… and when we g… and when we give children opportunities to respond.ive children opportunities to respond.

So we designed an intervention.So we designed an intervention.

Small groups of children participate in ‘listening Small groups of children participate in ‘listening

centers.’centers.’

Prerecorded storybooks

and explicit, embedded

lessons are delivered

under headphones.

MONDAY

TUESDAY

Multiple listens

provide repeated

exposure to

instruction and many

opportunities to

respond.

WEDNESDAY

Measures of

instructional content

are administered for

each book.

What skills should we target in our intervention?

Early Language and Literacy Early Language and Literacy DomainsDomains� Phonological Awareness

� Alphabet Knowledge

� Vocabulary

� Comprehension� Comprehension

Early Language and Literacy Early Language and Literacy DomainsDomains� Phonological Awareness

� Alphabet Knowledge

� Vocabulary

PA Intervention

� Vocabulary

� Comprehension VC Intervention

TIER 2 VOCABULARY TIER 2 VOCABULARY AND COMPREHENSION AND COMPREHENSION AND COMPREHENSION AND COMPREHENSION INTERVENTIONINTERVENTION

Background and RationaleBackground and Rationale

� Oral language skill in preschool, including vocabulary and comprehension, predicts later reading proficiency.

� Many children enter kindergarten with � Many children enter kindergarten with limited oral language skills that place them at risk for later reading difficulties.

� There is a need for curricula that can improve oral language skills in early childhood.

Child Outcomes“Business as Usual” Instruction

Proximal:

• Limited growth in oral language skill

Distal:

• Poor reading comprehension

• Increased chance for reading disability

• Low rates of explicit instruction in vocabulary and

comprehension

• Limited exposure to rich linguistic input

• Few practice opportunities

• Lack of consistent instructional language

• Limited instructional support provided by

curriculum

Tier 2

InterventionChild Outcomes

Proximal:

• Growth in oral language skill.

• Acquisition of vocabulary words

• Development of comprehension skills

• Increased metalinguistic skills

Distal:

• Improved reading comprehension

Supplemental Instruction

• High rates of explicit instruction in vocabulary

and comprehension

• Frequent active academic engagement

• Repeated practice opportunities in multiple

contexts

• Implemented with high fidelity

• Curriculum based assessments to monitor

progress

InterventionChild Outcomes

Intervention Component ExamplesExplicit Teaching

•Vocabulary words are presented with systematic instructional language in which words are emphasized, definitions are stated, information about words’ meanings are provided.

Enormous. Say enormous. Enormous means really big. Can you think of something that is enormous? What about…. a school bus! A mountain! Or a building! Those are things that are really big.

•Responses are provided after a pause for child’s response. ‘Think-aloud’ models of the evidence for the appropriate response are provided.

Why is Ellie happy? [pause for child response] Because she made new friends! I would be happy to have some new friends too.

Intervention Component ExamplesSelection of Appropriate Targets for Instruction

•Challenging vocabulary targets are selected to have high utility for academic achievement.

• Inferential questions are selected to facilitate reading comprehension.

protect, greet, selfish

Do you think the Jungle Friends will go to the beach again?

reading comprehension. to the beach again?

Why did Suki’s mom take her to the movies?

Intervention Component ExamplesMaximize Opportunities to Respond

•Opportunities to respond are maximizedwithin books

• 6 embedded lessons and 3 review lessons per book • 2 – 6 opportunities to respond per lesson

Ellie’s First Day: 16 opportunities to respond related to vocabulary, 3 opportunities to respond related to comprehension.

lesson

•Multiple intervention sessions provide repeated opportunities to respond and to practice responses

• 3 intervention sessions per book• 9 books per series

Intervention sessions on three days (e.g., Monday, Wednesday, Thursday).

Ellie’s First Day Leo’s Brave FaceJungle Friends Go

to the Beach

Vocabulary Words

enormous brave soaked

different grin gorgeous

ComprehensionQuestions

How do you think Ellie

feels about meeting

new friends? [Why?]

How do you think Leo

feels about going to the

dentist? [Why?]

What do you think will

happen in this story?

Where did Ellie go in What did Leo learn How did Tanisha feel Where did Ellie go in our story?

What did Leo learn from the dentist?

How did Tanisha feel when the wave knocked over her sandcastle?

At the end of the story, Ellie was happy. Why was Ellie happy?

At the beginning of the story, Leo was afraid of the dentist. What do you do when you are afraid?

Do you think theJungle Friends will go to the beach again? [Why or why not?]

The Forest Friends are thrilled! They are excited to go to the carnival. Thrilled. Say thrilled. (2) Thrilled means excited. Tell me, what word means excited? (2) Thrilled! Good work! When are you thrilled? (2) What about… when you get a present! …Or your friends come over to play! I bet that makes you feel excited. Now, lift the flap. Look! These boys are at a birthday party. They are excited. They are thrilled! Tell me, what does thrilled mean? (3) Excited! That’s right.

Do you think Marquez Monkey will have another backwards day? (3) Yes, I think he will! Marquez had so much fun doing everything backwards. He will probably want to do it again.

Video ExampleVideo Example

TIER 2 VC EARLY TIER 2 VC EARLY EFFICACY STUDYEFFICACY STUDY

2010-2011

EFFICACY STUDYEFFICACY STUDY

ParticipantsParticipants

� 9 preschool children in 3 classrooms were identified with limited oral language skills in fall.

� Multiple gating procedures for � Multiple gating procedures for identification that included a teacher survey, Picture Naming IGDI 2.0, norm-referenced tests.

Characteristics of ParticipantsCharacteristics of Participants

School Child Age GenderOral

Language PLD

Picture Naming 2.0

PPVT-IV CELF-P2

School A

A1 4;9 Female 1 6 80 86

A2 4;9 Male 2 10 78 88

A3 4;6 Male 3 7 88 77A3 4;6 Male 3 7 88 77

School B

B1 4;11 Male 1 5 83 73

B2 4;10 Female 2 9 87 86

B3 4;11 Female 3 6 96 90

School C

C1 4;10 Male 2 11 83 90

C2 4;5 Male 2 6 80 94

C3 4;3 Female 2 5 84 94

PPVT-IV: M = 84.3, Range 78 – 96; CELF-P2: M = 86.4, Range 73 - 94

MethodMethod

� Single-case repeated acquisition design

� Intervention was 9 books with embedded vocabulary and comprehension lessons.

� Measures:� Measures:

◦ Mastery monitoring probes at pretest and posttest for each book

◦ 2 outcomes: Vocabulary and Comprehension

Mastery Monitoring Items and Mastery Monitoring Items and ScoringScoring� Vocabulary◦ Maximum score at pretest and posttest was 6.

◦ One 3-point item per word� Open ended question scored 0, 2, 3� Open ended question scored 0, 2, 3

� "Tell me, what does enormous mean?“

� "Really big" 3 points

� “means a big building” 2 points

� “I don’t know” 0 points

� If child scores 0, receives the recognition question scored 0 or 1.� "Does enormous mean 'really big' or 'really cold'?“

� “Really big” 1 point

� “Really cold” 0 points

Results: VocabularyResults: Vocabulary

� Criteria for treatment effect:

◦ Pretest-posttest difference of at least 2

◦ Posttest score of at least 3

� Treatment effects for all participants for � Treatment effects for all participants for many books (Range: 2-8 books).

� Children learned an average of 7.1 words of the 18 possible (Range: 2-13).

2

3

4

5

6Participant A1

Pretest

Posttest

Mastery Monitoring Score

0

1

2

1 2 3 4 5 6 7 8 9

Pretest

Book

Mastery Monitoring Score

3

4

5

6

0

1

2

3

4

5

6

Pretest

Posttest

Participant A1

Participant C1

Mastery Monitoring Score for

Vocabulary

0

1

2

3

0

1

2

3

4

5

6

1 2 3 4 5 6 7 8 9

Participant B3


Vocabulary

Mastery Monitoring Items and Mastery Monitoring Items and ScoringScoring

� Comprehension

◦ Maximum score at pretest and posttest was 6.

◦ Three 2-point comprehension questions◦ Three 2-point comprehension questions

� "At the end of the story, Ellie is happy. Why is Ellie happy?“

� “Because she made new friends” 2 points

� “Because she likes playing” 1 point

� “Her big” 0 points

Results: ComprehensionResults: Comprehension

� Criterion for treatment effect:

◦ Pretest-posttest difference of at least 2

� Treatment effects for most participants for many books (Range: 0-6 books).for many books (Range: 0-6 books).

� Average gain score per book was 1.1 points (SD = 1.66, Range = -4-4).).

Participant A1

Participant C1


Comprehension

0

1

2

3

4

5

6

1 2 3 4 5 6 7 8 9

3

4

5

6

Participant B3


Comprehension

0

1

2

3

1 2 3 4 5 6 7 8 9

0

1

2

3

4

5

6

1 2 3 4 5 6 7 8 9

We have We have We have We have

learned a lot…learned a lot…

FindingsFindings

� Children are actively engaged in prerecorded storybooks and embedded lessons.

� Children learn many of the instructional � Children learn many of the instructional targets.

� High fidelity implementation is feasible.

…but there is still work to be done.

Questions that Remain….Questions that Remain….

� Why do children learn some words and not others?◦ Words are too hard? Definitions are abstract? Interventions are distracting? Stories aren’t engaging?engaging?

� Are kids retaining vocabulary words?

� Are we teaching comprehension skills or how to answer questions about a particular story?◦ Study design that is a more rigorous test? A comprehension progress monitoring measure?

Questions?

Tier 2 Phonological Tier 2 Phonological Awareness InterventionAwareness Intervention

RTI Summit: Monday, September 26th 2011

Sean Noe &

Trina Spencer

BackgroundBackground

�Phonological awareness (PA) is strongly correlated with reading success

�Unfortunately, many children from low SES backgrounds struggle to learn important early backgrounds struggle to learn important early literacy skills like PA

�Storybook reading may be a useful means of providing important supplemental instruction

Overview of PA interventionOverview of PA intervention

� 12 books

� Pre-recorded scripts

� Embedded interventions

◦ Instruction on PA skills◦ Instruction on PA skills

Arrangements and MaterialsArrangements and Materials

� Small-group listening centers outside of class.◦ Maximum of 3 children + 1 adult facilitator

� Each child and facilitator has a book.

� Children listen to the prerecorded story with embedded lessons on headphones.embedded lessons on headphones.

� The facilitator…◦ acts as a child participant

◦ encourages desired behavior

◦ discourages undesired behavior

◦ models appropriate responses, but provides no additional information

PA vs. VC InterventionPA vs. VC Intervention

� VC: each book contains instruction on specific, independent words◦ Listen to same book on consecutive days◦ Each book’s content is independent

� PA: skills are closely relatedPA: skills are closely related◦ Use multiple exemplar training to teach…� metalinguistic skills of sound awareness and manipulation� as opposed to specific examples (e.g., cat rhymes with hat)

� Differences◦ Repetition of concepts not the same items◦ 1,2,3,1,2,3 instead of 1,1,1,2,2,2,3,3,3◦ Books read twice instead of three times

Description: 12 Books

MONDAY

TUESDAY

WEDNESDAY

Description: Sequence of Books

THURSDAY

FRIDAY

MONDAY

Two cycles provide repeated exposures to instruction and many opportunities to respond.

Description: Embedded LessonsDescription: Embedded Lessons

� Before Story ◦ Introduction to the task

◦ Review of a letter and its sound

� During Story ◦ 4 embedded lessons (aka “interruptions”)◦ 4 embedded lessons (aka “interruptions”)

◦ Multiple opportunities for practice

◦ Systematic scaffolding� Support is faded across interruptions

� After Story◦ 1 embedded lesson

◦ Practice of skill without support (feels like a test)

Description: Instructional TargetsDescription: Instructional Targets

Embedded lessons with explicit instruction are provided for…

� Blending (unit 1)

Segmenting (unit 2)� Segmenting (unit 2)

� Identifying First Sounds (unit 3)

� Rhyming (unit 4)

How we got hereHow we got here

� Previous study ◦ 6 rhyming books

◦ 6 alliteration books

◦ Read 3 times each on consecutive days

� Issues◦ Kids tried to memorize specific examples◦ Kids tried to memorize specific examples

◦ Kids focused on semantics of word, couldn’t generalize beyond specific examples◦ Kids scored high on mastery monitors, but couldn’t perform on generalization measures

� Revisions◦ New order of books

◦ Teach simplified blending and segmenting skills

◦ Use of non-sense words

◦ Added activities to help focus their attention on the sounds

Description: Scope and SequenceDescription: Scope and Sequence

Unit

Book 1 MMA 2 ECF 3 LHR 4 NJF 5 EGS 6 MBD 7 PPP 8 SDF 9 FSS 10 BEV 11 SSSu 12 PMM

segmenting 2

phonemes

segmenting 3

phonemes

segmenting 2/3

phonemes

blending 3

phoneme with

final stop (fit)

first sound

(continuous)

Letter Reviewed P M R L F B S D T C

rhyme production

blending 2-syllable words

(painting)listening for sounds in words sound matching first sound identification (stop)

blending 3 phoneme continuous

sounds (man)first sound identification rhyme matching

Phonological Awareness

Unit 1 Unit 2 Unit 3 Unit 4

Skills Taught

blending compound words

(football)

blending and segmenting 3 phoneme stop sounds

(nonsense)

Embedded

Lessons

Let's put a first sound on __ to make some silly

words. Now look at the picture and find

something that ends with __.

Tell me a word that ends with

__.

Activity Stretch and clapSlide your finger on green line between two or

three circleDoors & initial sound song

Tapping boxes

Handprints

Instructional

Language

Listen to me say __ the slow way. Now listen to

me say __ the fast way. Say __ with me the fast

way. Say __ the fast way.

Listen to me say __ the slow way. Say __ with me

the slow way. Say __ the slow way.What's the first sound you hear in __?

Say __ with me the slow way while you tap the

boxes.

Raise your hand when you hear

__ (sound).

The

first/middle/

last sound you

hear in __ is _.

What's the first

sound you hear

in __?

Let's say some words the slow way to find the

one that starts with __. Which one starts with

__?

DESCRIPTION: SAMPLE LESSON AND INSTRUCTIONAL DESCRIPTION: SAMPLE LESSON AND INSTRUCTIONAL LANGUAGELANGUAGE

Our embedded lessons have consistent instructional language.

The friends thought running up a slide was a silly thing to do.

But since it looked like fun, they decided to try it, too.

This includes…

�A regular pattern of introducing tasks and teaching PA skills

� Multiple opportunities for children to respond.

�Ample pause times to ensure children have the chance to respond.

Audio: Bark bark. The Jungle Friends are walking up the slide backwards. Do you think they might fall? Say fall with me the slow way: fffooolll. Put your finger in the blue circle and leave it there. What’s the first sound you hear in fall? (1) /f/. The first sound you hear in fall is /f/. Great job!

Summer 2011 StudySummer 2011 Study

ParticipantsParticipants

� Children attended a daycare program that serves low-income families

◦ 9 children in 3 groups of 3 children

� 3 children served as target participants based � 3 children served as target participants based on…

◦ consistently low scores on pretest and baseline curriculum based measuresChild Age Gender WPF TOPEL-PA

Tammy 4.4 F 1 90

Laura 3.4 F 0 89

Joe 4.7 M 0 76

Study Design and MeasurementStudy Design and Measurement

� Multiple probe across participants

◦ Staggered introduction of instructional targets (Units)

� Measurement� Measurement

◦ After reading 3 books on consecutive days, we probed progress using curriculum based measures

� Blending (OH)

� Segmenting (OH)

� Rhyme Production (OH)

� First Sound Fluency (DIBELS)

◦ Word Parts Fluency (DIBELS) � Pre- and Post-test

CurriculumCurriculum--Based MeasuresBased Measures

� Blending

◦ Listen: Flag…pole. Now say it the fast way. (flagpole)

� Segmenting� Segmenting

◦ Listen: Man. Now say it the slow way. (mmmaaannn)

� Rhyming

◦ Tell me a word that ends with AT. (cat)

� First Sound Fluency

◦ What is the first sound you hear in the word toad? (/t/)

TammyTammy

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 100

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10

Blending Score

Segmenting Score

Baseline

Baseline

InterventionIntervention

Begin to focus on

segmenting

CBM ProbesCBM Probes

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10

First Sound Fluency Score

Rhyme Production Score

BaselineBaselin

e

InterventionIntervention

Begin to focus on first sounds and alliteration

Introduce concept of first sounds

Begin to focus on rhyming

CBM Probes CBM Probes

LauraLaura

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Blending Score

Segmenting Score

CBM ProbesCBM Probes

Begin to focus on

segmenting

Baseline

Baseline

Intervention Intervention

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10 11 12 13 14

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14






Baseline

Baseline


JoeJoe

0

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Blending Score

Segmenting Score


Baseline

Baseline


Begin to focus on

segmenting

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10 11 12 13 140

1

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14



CBM Probes


CBM Probes

Baseline

Baseline




10

12

14

16

18

20

Word Parts Fluency

18

16

14

12

10

(Tammy)

WPF Score

0

2

4

6

8

10

Pretest Posttest

8

6

4

2

0(Laura & Joe)

WPF Score

Challenges and SolutionsChallenges and Solutions

� Challenge

◦ Selection of participants

� Content and performance

� Learning and behavior� Learning and behavior

� Age and exposure to curriculum

� Solution

◦ Use IGDI’s (Rhyming and Alliteration) as primary PA screening measures in Gate 1

◦ Add teacher questionnaire as Gate 2

◦ Consistent low performance in baseline as Gate 3

Teach QuestionnaireTeach Questionnaire Child __________________Child __________________

Directions: Consider the following statements as they relate to [childDirections: Consider the following statements as they relate to [child’’s name]. Make a s name]. Make a check in the accompanying box if you feel the statement is true.check in the accompanying box if you feel the statement is true.

Statement

(1) This child often needs extra time in order to learn new things.

(2) This child often requires repeated instruction to learn a new skill.

(3) This child often needs to have activities simplified or modified.(3) This child often needs to have activities simplified or modified.

(4) This child’s behavior often interferes with his/her learning.

Statement

(5) This child knows some letter sounds.

(6) This child is able to participate independently at centers.


� Challenge

◦ Timing of intervention

� Fall – children are young and haven’t been exposed to literacy curriculumliteracy curriculum

� Spring – might leave less time for intervention

� Solution

◦ Conduct a comparison between Fall and Spring participants who received the intervention.


� Challenge

◦ PA skills are closely related

� Difficult to establish experimental control in SSR when carryover is expected and desiredcarryover is expected and desired

� Solution

◦ Conduct a randomized control group design

� Pre/Posttest measures

� Randomized Control Group Design

� Approximately 84 children across 4 sites and 2 time points

◦ Must meet selection criteria

◦ Use information about content, learning, and behavior

� Randomly assigned to condition

◦ Half will be assigned to treatment condition

Revised StudyRevised Study

◦ Half will be assigned to treatment condition

◦ Half will be assigned to control condition

� Fall vs. Spring comparison

◦ Half will receive intervention in Fall of 2011

◦ Half will receive intervention in Spring of 2012

� Treatment group will read intervention books for 24 days

� Control group will receive ‘business-as-usual’

� Both groups will receive pre-test and post-test measures

Challenges we are still resolvingChallenges we are still resolving

� Challenge

◦ Measurement

� Preschoolers are not reliable test takers

� Test task or concept� Test task or concept

� Possible Solution

◦ Test less often

◦ Provide more sample items

◦ Only test one task a day

Challenges we are still resolvingChallenges we are still resolving

� Challenge

◦ Instructional Aim

� Rhyming and Alliteration

� Syllable and word level segmenting� Syllable and word level segmenting

� Phoneme level segmenting

�FSF benchmark at the beginning of kindergarten is 10

� Possible Solutions

◦ Increase instructional focus within books on first sounds

◦ Increase repetitions of unit 3 books

◦ Create an additional set of books with a first sound

Questions and DiscussionQuestions and Discussion

Development and Evaluation of Tier 2 Instruction to ...€¦ · Elizabeth Spencer Howard Goldstein...

Documents

Transcript of Development and Evaluation of Tier 2 Instruction to ...€¦ · Elizabeth Spencer Howard Goldstein...