A New Era of Marketing - Integrating Direct and Digital Marketing
Integrating learning theory in an era of accountability testing
21
Copyright © 2007 Educational Testing Service Integrating learning theory in an era of accountability testing Drew Gitomer ETS Conference on Educational Testing in America: State Assessment, Achievement Gaps, Federal Policy and Innovations September 8, 2008
description
Integrating learning theory in an era of accountability testing. Drew Gitomer ETS Conference on Educational Testing in America: State Assessment, Achievement Gaps, Federal Policy and Innovations September 8, 2008. The Problem We Are Confronting. - PowerPoint PPT Presentation
Transcript of Integrating learning theory in an era of accountability testing
Copyright © 2007 Educational Testing Service
Integrating learning theory in an era of
accountability testingDrew Gitomer
ETS
Conference on Educational Testing in America: State Assessment, Achievement Gaps, Federal Policy and Innovations
September 8, 2008
2
The Problem We Are Confronting
• Accountability assessments grounded in an outdated scientific model for conceptualizing proficiency, teaching it, and assessing it
• Accountability is based on a limited set of limited proxies
• Interim assessments, formative assessments, and teacher professional development that are emulating the less desirable characteristics of those accountability tests
(aka the irony of alignment)
3
The Vision
Can have an assessment system that contributes to educational practice by:
• Documenting what students have achieved (“assessment of learning”),
• Helping identify how to adjust instruction (“assessment for learning”), and
• Engaging students and teachers in worthwhile educational experiences in and of themselves (“assessment as learning”)
4
Coherence of Three Components
• Public accountability functions
• Formative assessment
• Professional development
What is coherence?
5
Current Focus of the Project
• Reading, writing and mathematics
• Middle school
• Primary work in Maine (Portland)
• Beginning work in other places
• Today’s talk focuses on reading and the work cited is led by Kathy Sheehan and Tenaha O’Reilly
6
The Foundation: A Strong Conceptual Base
• Built on:– Cognitive-scientific research
• Principles
• Domain-specific competency models
• Developmental models as available
– Aligned with State standards
7
The Reading Competency Model: The Skills Branch
Understand & Use Texts
Prerequisite Reading Skills
Basic Comprehension Skills
Applied Comprehension Skills
Pre-Reading Strategies
Model Building Strategies
Strategies for Going Beyond the Text
Required Skills
Reading Strategies
Text Conv. & Characteristics Persuasive Text
Informational Text
Literary Text
Learning to Read
Reading to Learn
Reading to Do
8
The Reading Competency Model: The Strategies Branch
Understand & Use Texts
Prerequisite Reading Skills
Basic Comprehension Skills
Applied Comprehension Skills
Pre-Reading Strategies
Model Building Strategies
Strategies for Going Beyond the Text
Required Skills
Reading Strategies
Text Conv. & Characteristics Persuasive Text
Informational Text
Literary Text
Deliberate, conscious, effortful actions that readers can implement to repair breaks in comprehension & enhance understanding.
Ask/answer pre-reading questions
Use Graphic Organizers
Adopt a Critical Stance
9
The Reading Competency Model: The Text Branch
Read
Prerequisite Reading Skills
Basic Comprehension Skills
Applied Comprehension Skills
Pre-Reading Strategies
Model Building Strategies
Strategies for Going Beyond the Text
Required Skills
Reading Strategies
Text Conv. & Characteristics Persuasive Text
Informational Text
Literary Text
Knowledge of conventions that contribute to reading ease or difficulty
10
The Text Branch
Difficulty Drivers
The Text
Genre
Informational
Research/Persuasive
Literary
Vocabulary
Syn. Complexity
Cohesion
Structure, Negation, etc.
(Based on Sheehan, Kostin & Futagi, in press)
Ensure that: (1) targeted skills are assessed within a grade-appropriate span of text complexity; and (2) assessment tasks focus on those aspects of text variation that are likely to have a significant impact on reading performance.
11
The Skills Tested by Most Existing Reading Comprehension Tests
Recognize Common Words
Integrate Inf w Bk. Know.
Integrate Inf. Across Texts
Evaluate/Critique
Prerequisite Reading Skills
Basic Comprehension
Skills
Applied Comprehension
Skills
Decode Unfamiliar Words
Read Fluently
Locate/Retrieve Inf.
Und. Diff. Voc./Lit. Term.
Infer Relations Amg Concepts
Und. & Use Discourse Struc.
Understand & Use Texts
12
Key Features - Reading Design
• Measure the full range of skills, including prerequisite skills
• Purposeful, authentic, scenario-based tasks
• Support reading across the curriculum– Informational Texts: Excerpts from the introductory chapters of
science and social studies textbooks, newspaper & magazine articles, etc.
– Persuasive Texts: Editorials, Letters to the editor– Literary Texts: Narratives, memoirs, poems
• Select task formats that encourage use of targeted reading strategies
• Include a mixture of expressive (CR) and receptive (mc) tasks
• Minimize production requirements– Want to test reading not writing
13
Implementation: The Assessment Design
• Period accountability assessments (PAAs)
• Foundational tasks– Constructed response and selected response
– Scaffolded structure
• Supplemental tasks
• Computer Administration
• Automated Scoring
14
CBAL vs. (stereo)typical assessments
Traditional CBAL
Single Measurement Occasion
Multiple Measurement Occasions
Many short items A few long tasks
(mostly) unrelated Centered around a common theme
Representative of a domain
Based on a Competency Model
Homogeneous Response Types
Heterogeneous Response Types
15
A Sample PAA
• Spoken Module– Provide mastery evidence relative to the Prerequisite Skills– Short Tasks: Word Recognition, Decoding, Oral Fluency– Students speak into a headset, responses are scored as
Words Correct per Minute (wcpm)• Comprehension Module
– Provide mastery evidence relative to the Model-Building & Applied Comprehension Skills
– All texts & tasks revolve around a meaningful scenario• e.g., You have to write a report about the scientific method
for your science class– Tasks employ innovative response formats: SR and CR– Multi-step thinking, scored via partial credit scoring rubrics
16
The Scenario
• Four Texts– A passage about Ben Franklin adapted
from a Social Studies textbook
– An article about the Scientific Method extracted from an Encyclopedia
– A newspaper article about 3 winners of the Intel Science Competition
– A diagram from a student lab report
You have to write a report about the Scientific Method for your science class. Since you enjoy reading about American History, you decide to focus your report on Ben Franklin’s use of the Scientific Method. Read the following passages to learn more about the Scientific Method and about Ben Franklin’s scientific experiments.
17
Preliminary Results – Construct Validation
TOWRE Gates z’
The Spoken
Module
0.78
0.63
2.69
**
The Comprehension
Module
0.46
0.79
5.18
***
TOWRE: a standardized measure of Prerequisite Skills Gates-MacGinitie: a std. measure of Model-Building Skills
18
The Assessment is Also Designed to Yield Diagnostic Information that Teachers can Use to Adjust Instruction
Word Callers
Gap Fillers
CBAL Spoken Module Score (wcpm)
P(c
orr
ec
t |
Lit
era
l C
om
pre
he
ns
ion
)
0 20 40 60 80 100 120 140 160
0.0
0.2
0.4
0.6
0.8
1.0
Word Callers
Gap Fillers
CBAL Spoken Module Score (wcpm)
P(c
orr
ec
t |
Lit
era
l C
om
pre
he
ns
ion
)
0 20 40 60 80 100 120 140 160
0.0
0.2
0.4
0.6
0.8
1.0
19
Formative System
• Is built upon the same conceptual base as the Accountability System
• Gets information from the Accountability System
• Is used at the district’s and teacher’s option
• Is available on demand
• Performance in the formative system will not be used for accountability purposes
20
Structure of formative assessment
• Library of resources including professional development component– Elaborated foundational tasks – for each task
component• Clarification of learning intentions – connection to
competency model
• Criteria for considering student responses
• Common conceptual challenges
• Potential pedagogical strategies
• Potential additional resources
21
Challenges
• Psychometric
• Automatic scoring
• Educational infrastructure
• Political accountability
