Three Self-Evident, but Heretical, Truths about Reading Assessment

Michael D. BeckQuestar Assessment, Inc.

mbeck@questarai.com

1. Students evidence “reading” NOT by

reading a text, but by answering questions.

1. Students evidence “reading” NOT by silently reading a text, but by answering questions.

Text complexity is important. However, it’s only one of two critical factors determining a

student’s “reading achievement.”

Readability formulas are inaccurate and misleading indicators of reading level unless item difficulty is taken into account and matched with the text level.

The Relationship between Text & Item Difficulty

TEXT A:• Grade 4 lengthy

(narrative) text• 1400 words +/-• “Readability” =

Lexile approx. 700-780 Dale-Chall Gr. 5

ITEMS:

• 7 M-C items• 3 “literal”, 4 “inference”• p-value range .47-.88• Median p-value = .67• Median point-biserial - 47

The Relationship between Text & Item Difficulty

TEXT B:• Grade 4 lengthy

(narrative) text• 1400 words +/-• “Readability” =

Lexile approx. 700-780 Dale-Chall Gr. 5

ITEMS:• 6 M-C items• 3 “literal”, 3 “inference”• p-value range .39 - .63• .39, .45, .47, .50, .54, .63• Median p-value = .48

• Median pt-biserial = .44

The Relationship between Text & Item Difficulty

TEXT C:• Grade 4 lengthy

(narrative) text• 1400 words +/-• “Readability” =

Lexile approx. 700-780 Dale-Chall Gr. 5

ITEMS:• 6 M-C items• 3 “literal”, 3 “inference”• p-value range .71 - .94• .71, .77, .83, .88, .90, .94• Median p-value = .86

• Median pt-biserial = .43

Assessing “Reading Level”Item sets – all “good” items. All texts are “Grade 4/5” level.

Text p-values (Gr. 4 midyear) mdn mdn pb A 47, 50, 54, 63, 71, 83, 88 63 .47 B 39, 45, 47, 50, 54, 63 48 .44 C 71, 77, 83, 88, 90, 94 86 .43

Which of these 4th - 5th grade passage(s) can:

“3rd grade readers” read ? “4th grade readers” ?

Text – Item SummaryText p-values (Gr. 4 midyear) median A 47, 50, 54, 63, 71, 83, 88 63 B 39, 45, 47, 50, 54, 63 48 C 71, 77, 83, 88, 90, 94 86

Which of these 4th / 5th grade passage(s) can: “3rd grade readers” read ? “4th grade readers” ?

GUESS WHAT?

Text – Item Summary

• These are all the same passage.

• Same passage tried with >20 items

• Median p-value: subset A = .63 subset B = .48 subset C = .86

Text – Item Summary• These are all the same passage.

• Median p-value: A = .63 B = .48 C = .86

What “controls” the difficulty ?What is the “difficulty” of this text?

0%10%20%30%40%50%60%70%80%90%

100%

Gr 2 Items Gr 3 Items Gr 4 Items Gr 5 Items Gr 6 Items

Gr. 5+ Gr. 4 Gr. 3 Gr. 2 - 2 Gr. 2 - 1 Gr. 1 - 2

Grade 3 Philadelphia Public SchoolsInstructional Reading Level—Validation of Instructional Reading Levels

Let’s be careful• Common Core Standards (Appendix A)

improperly overstates the import of text difficulty.

• Don’t assume you take care of “difficulty” by taking care of the text.

• Eschew enchantment with “readability” as a direct indicator of reading level

2. To gauge childrens’ reading skills, it is more important to know their

instructional reading level than to actually measure

these reading skills.

2. We better gauge reading skills from students’ IRLs than by actually

measuring these skills.

• An IRL provides a better estimate of reading skills achievement than does a measure of these reading skills. (no kidding)

• Analogy to assessing “writing”

0%10%20%30%40%50%60%70%80%90%

100%

Consonants Vowels Vocabulary Word Parts

Gr. 5+ Gr. 4 Gr. 3 Gr. 2 - 2 Gr. 2 - 1 Gr. 1 - 2 -

Grade 3 Philadelphia Public SchoolsInstructional Reading Level—Validation of Skills Assessment

Validating IRLs as Evidence of Reading Skills Performance

Skill / Teaching Level Percent of Students by IRL (Grades 5 & 6) Gr2 Gr3 Gr4 Gr5 Gr6 Gr7-8 Gr9+Phonics: Consonants Introduce 77% 53% 35% 9% 3% 0% 0% Practice 0 36 42 36 20 8 4 Apply 23 11 23 55 77 92 96Phonics: Vowels Introduce 55 84 55 19 5 0 0 Practice 10 15 31 36 21 8 3 Apply 35 1 14 45 74 92 97Structural Analysis Introduce 99 98 83 43 9 1 0 Practice 1 1 16 48 56 26 9 Apply 0 1 1 9 35 73 91Skimming & Scanning Introduce 100 96 92 70 34 8 3 Practice 0 4 8 26 49 36 18 Apply 0 0 0 4 17 56 80

<1%

9% 10%

6

33

3

31

6

<1

4%

12% 11%

10

29 4

23

7

1

Philadelphia – Gr.3 Data – Skills Tests vs. IRLs& Teacher Judgment vs. IRLs

Skills Need based on IRL Intro Pract. Apply

Skills Need based onActual Skills Test

Skills Need based onClassroom Teacher Judgment

Apply

Pract.

Intro

Apply

Pract.

Intro

The analogy to assessing “writing”How we typically assess “real” writing (Method W)

How “writing skills” are typically assessed (Method S)

Which method is more reliable? BUT, is it valid?

Which method is more generalizable? Which method correlates highest with “true writing ability” – defined as constructed writing of multiple (>5) products of various lengths, scored by trained teachers with a rubric?

Which method correlates highest with a different measure of “true writingability” – defined as “scores” from 1-6 assigned by classroom HS Englishteachers at the end of the school year?

The analogy to assessing “writing”How we typically assess “real” writing (Method W)

How “writing skills” are typically assessed (Method S)

Which method is more reliable? Of course, Method S.BUT, is it valid?

Which method is more generalizable? Method S

Which method correlates highest with “true writing ability” – defined as free writing of multiple (>5) products of various lengths, scored by trained teachers with a rubric? Method S

Which method correlates highest with a different measure of “true writingAbility” – defined as “scores” from 1-6 assigned by classroom HS EnglishTeachers at the end of the school year? Method S

What’s the Point ?• Reading skills (and “writing”) can be

assessed with high reliability, generalizability, and validity. We just rarely do it.

• We can judge (estimate) a student’s reading skills better from a “solid” IRL than from a semi-reliable skills test.

• Why test skills, unless we’re going to do a better job?

3. In U.S. education history, a total of 3.14

teachers have ever said:

3. In U.S. education history, a total of 3.14 teachers have

ever said: more assessments –

“formative” or otherwise – would help

me teach better.

Stipulations• I like tests, very much.• I like – and respect greatly - teachers.• I like state assessment offices and folks.• I even like “formative” assessment.• Some districts/buildings/teachers CAN use

“classroom assessment” well. Rick Stiggins can name every one – and count them on his digits.

3. Cram-down formative assessments will NOT improve

education.• Stop the lie• Any instrument emanating from a state (or

national) capital is, by definition, instructionally inert.• Stop wasting precious goodwill & $

3. Cram-down formative assessments will NOT improve

education.• If we really care, preservice must change.

• Give 2, 3 pieces of semi-useful information on the statewide accountability test, then get out of the classroom.

• Then if we really care a lot, improve the wasteland that we now define as “professional development.”

If we knew what we were doing, it wouldn’t be called research, would it?

Albert Einstein