Designing Math and ELA Tasks for AA-AAS Using ECD and UDL
description
Transcript of Designing Math and ELA Tasks for AA-AAS Using ECD and UDL
This material is based on work supported by two Enhanced Assessment Grants from the U.S. Department of Education: Alternate Assessment Design—Mathematics and English Language Arts. The PADI online system was developed through a grant from the National Science Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the U.S. Department of Education or the National Science Foundation.
Designing Math and ELA Tasks for AA-AAS Using ECD and UDL
National Conference on Student AssessmentNational Conference on Student AssessmentJune 19, 2011June 19, 2011
Renée Cameto and Geneva Haertel – SRI InternationalWendy Carver – Utah State Office of EducationKaren Denbroeder – Florida Department of EducationCarol Scholtz – Idaho State Department of EducationDeborah Matthews – Kansas State Department of Education
Shawnee Wakeman – ModeratorEdward Roeber – Discussant
Introduction Based on work from two Enhanced
Assessment Grants funded by the U.S. Department of Education
Each project involved a consortium of states and SRI International: Alternate Assessment Design–Mathematics
Utah, Idaho, Florida Alternate Assessment Design–English
Language Arts Idaho, Utah, Kansas
2
Agenda ECD Framework Co-Design Process Example Design Patterns and Tasks Task Tryouts
Teacher training and materials Videos of task administration Results
Panel Discussion with State Representatives Q & A Discussant
3
Need Alternate Assessments need to be well
designed The design and development processes
employed need to be systematic, rigorous, and reflective of industry standards
NCLB/ESEA require that students with disabilities, including SWSCD, be assessed on grade-level academic content
Assessment continues to be a lever of educational change (assessment drives instruction and school reform)
4
Innovative Approach Evidence-centered design (ECD) is an
innovative assessment design process Initial work on ECD conducted at ETS by Mislevy,
Steinberg, and Almond (2003) Has been used for more than 15 years; in these EAG
projects, ECD is being extended to the population of students with significant cognitive disabilities
Is a framework and set of processes Meets criteria for developing well-designed
assessments Supports the design of items that are aligned to the
focal constructs of interest
5
Innovative Approach ECD can be applied to:
All subject areas All grade levels All types of assessments (large scale, summative,
formative, technology-enabled, pencil/paper, advanced placement, workforce, etc.)
All types of alternate assessments (portfolio, performance task, checklist, etc.)
All item/task formats (multiple choice, constructed response, performance task, etc.)
Integrating UDL into the ECD framework promotes accessibility of items through consideration of student needs and abilities during initial design and throughout the design process
6
What is Evidence-Centered Design?
Critical Question: How do we judge what students know and what they can do? Make explicit what knowledge or skills are the target Identify non-target but required knowledge and skills Generate the kinds of observations that provide
evidence of a student having the target knowledge Determine the kinds of stimuli and work products
needed to gather the evidence These attributes form the basis of the assessment
argument - an argument based on evidence
7
Co-Design Process ECD process makes
use of co-design at all layers
Co-design team typically includes: Special educators
(experience with SWSCD)
Content experts Assessment specialists State Department of
Education administrators
8
Alternate Assessment Design
A multistep process using ECD co-design
9
Select Standards
Common Core
Domain Analysis
Create Design Patterns
Domain Modeling
Create Summary Task
TemplatesConceptu
al Assessme
nt Framewo
rk
Author Assessment
Tasks
Assessment Implementati
on
Pilot and Refine Tasks
Assessment Delivery
Select StandardsGOAL: In Domain Analysis, content relevant to the assessment is organized and selectedIdentify standards to meet state AA-AAS needs, for example, by identifying commonalities among the consortium states’ extended standards and comparison standards (e.g., NCTM Standards and Expectations, Common Core State Standards)Develop list of standards common to all participating states
10
Select Standards
Common Core
Create Design PatternsGOAL: In Domain Modeling, a narrative description of the assessment argument structure (Design Pattern) is produced to guide task developmentDesign Patterns for selected standards are developedDesign Patterns are reusable and improve efficiency of task developmentDesign Patterns can improve content validityDesign Patterns create a design space for assessment developers
11
Create Design
Patterns
Elements of Design Patterns
Student Model What knowledge, skills or abilities
(KSAs) should be assessed?
What construct (complex of student attributes) should be assessed?
Focal KSAs
Primary Knowledge, Skills, and Abilities (KSAs) targeted by this DP
Additional KSAs
Other KSAs that may be required by tasks from this DP, some of which can be supported by UDL and accommodations
Educational Standards
Associations with Educational Standards from different states as well as national standards, if desired
12
Elements of Design Patterns
Student Model What knowledge, skills or abilities
(KSAs) should be assessed?
What construct (complex of student attributes) should be assessed?
Focal KSAs Primary Knowledge, Skills, and Abilities (KSAs) targeted by this DP
Additional KSAs
Other KSAs that may be required by tasks from this DP, some of which can be supported by UDL and accommodations
Educational Standards
Associations with Educational Standards from different states as well as national standards, if desired
13
Elements of Design Patterns
Evidence Model What behaviors or performances should
reveal the knowledge, skills, and abilities?
What behaviors should reveal the construct?
Potential Observations
Observed behaviors of students that can provide evidence of Focal KSAs
Potential Work Products
What students say, do, or make that provides evidence about the Focal KSAs
Potential Rubrics Some evaluation techniques that may apply
14
Elements of Design Patterns
Evidence Model What behaviors or performances should
reveal the knowledge, skills, and abilities?
What behaviors should reveal the construct?
Potential Observations
Observed behaviors of students that can provide evidence of Focal KSAs
Potential Work Products
What students say, do, or make that provides evidence about the Focal KSAs
Potential Rubrics
Some evaluation techniques that may apply
15
Elements of Design Patterns
Task Model What tasks, situations, or stimuli should
elicit those behaviors and performances?
What tasks should elicit those behaviors?
Characteristic Features
Aspects of assessment situations likely to evoke the desired evidence
Variable Features Aspects of assessment situations that can be varied in order to control difficulty or target emphasis on various aspects of KSAs
16
Create Design Patterns Task Model
What tasks, situations, or stimuli should elicit those behaviors and performances?
What tasks should elicit those behaviors?
Characteristic Features
Aspects of assessment situations likely to evoke the desired evidence
Variable Features
Aspects of assessment situations that can be varied in order to control difficulty, target emphasis, or provide support for Additional KSAs
17
Create Design Patterns Variable Features are the aspects of the
task that are varied to support the Additional KSAsUDL Category
Example Additional KSA
Example Variable Feature
Perceptual (Receptive)
Ability to perceive linguistic components of the stimulus material and question
Delivery mechanisms by which question is perceived (e.g., read aloud, concrete objects, Braille)
Skill and Fluency (Expressive)
Ability to communicate response (e.g., respond verbally, by using pictures, by making a selection from a group)
Response mode options (e.g., pointing, speech and verbalization, assistive device/augmentative communication)
Language and Symbols
Ability to decode text, symbols, or images
Level of abstraction required of student (e.g., concrete objects, images, text)
18
Create Design Patterns Variable Features are the aspects of the
task that are varied to support the Additional KSAs
UDL Category Example Additional KSA
Example Variable Feature
Cognitive Ability to process multistep problems
Provide graphic organizers
Executive Ability to plan and sequence
Prompts and scaffolds to estimate effort, resources, and difficulty
Affective Ability to engage (e.g., task-specific motivation)
Enhance relevance, value, and authenticity of tasks
19
Source: Center for Applied Special Technology (CAST), http://CAST.org
Author Assessment Tasks
GOAL: In Assessment Implementation, tasks are authoredInformation pre-populates from the Design Pattern to the Task TemplateTemplates are used to author a family of items aligned with a Focal KSA
Complexity is varied within the family of items
UDL is incorporatedItems individually administered
20
Author Assessment
Tasks
Author Assessment Tasks
Step 1. Pre-populate Task Template with information from the Design Pattern Focal KSAs and Additional KSAs Potential Observations and Work Product Variable Features for Cognitive Background and UDL Characteristic Features
Step 2. Select attributes for authoring items Focal KSAs and Additional KSAs Potential Observations and Work Products Variable Features for Cognitive Background and UDL
21
Author Assessment Tasks
Step 3. Select Variable Features to be used in the item family Indicate how Variable Features will be
applied (e.g., limit numbers to those with two or fewer digits, include multiple representations of stimulus materials)
Consider ways to vary complexity Consider ways to incorporate UDL
22
Author Assessment Tasks
Step 4. Create Item DirectiveItem 1 Item 2 Item 3a/3b
Examiner shows student a picture of three pizzas and says, Here are three pictures of parts of a pizza. Examiner then presents student with a note card with the numeric fraction “¾” and says, Which picture shows three fourths of a pizza?
Examiner presents students with two photos, one with a whole pizza and the other with a half of a pizza and says, Here are two photos of pizzas. Examiner then presents student with a note card with the numeric fraction “½” and says, Show me which photo shows half of a pizza.
3a) Examiner presents to student an illustration of two unlabeled pizzas and says, Here are two photos of pizza, a whole pizza and part of a pizza. Show me the photo that is a part of a pizza.3b) If student does not respond or responds incorrectly, examiner removes all stimuli but the photo of part of a pizza and says, [Look at/touch] the photo of part of the pizza.23
Mathematics Item 1 Examiner shows student a picture of three pizzas
and says, Here are three pictures of parts of a pizza.
Examiner then presents student with a note card with the numeric fraction “¾” and says, Which picture shows three fourths of a pizza?
24
¾
Author Assessment Tasks
Step 4. Create Item DirectiveItem 1 Item 2 Item 3a/3b
Examiner shows student a picture of three pizzas and says, Here are three pictures of parts of a pizza. Examiner then presents student with a note card with the numeric fraction “¾” and says, Which picture shows three fourths of a pizza?
Examiner presents students with two photos, one with a whole pizza and the other with a half of a pizza and says, Here are two photos of pizzas. Examiner then presents student with a note card with the numeric fraction “½” and says, Show me which photo shows half of a pizza.
3a) Examiner presents to student an illustration of two unlabeled pizzas and says, Here are two photos of pizza, a whole pizza and part of a pizza. Show me the photo that is a part of a pizza.3b) If student does not respond or responds incorrectly, examiner removes all stimuli but the photo of part of a pizza and says, [Look at/touch] the photo of part of the pizza.25
Mathematics Item 2 Examiner presents student with two photos, one
with a whole pizza and the other with a half of a pizza and says, Here are two photos of pizzas.
Examiner then presents student with a note card with the numeric fraction “½” and says, Show me which photo shows half of a pizza.
26½
Author Assessment Tasks
Step 4. Create Item DirectiveItem 1 Item 2 Item 3a/3b
Examiner shows student a picture of three pizzas and says, Here are three pictures of parts of a pizza. Examiner then presents student with a note card with the numeric fraction “¾” and says, Which picture shows three fourths of a pizza?
Examiner presents students with two photos, one with a whole pizza and the other with a half of a pizza and says, Here are two photos of pizzas. Examiner then presents student with a note card with the numeric fraction “½” and says, Show me which photo shows half of a pizza.
3a) Examiner presents to student an illustration of two unlabeled pizzas and says, Here are two photos of pizza, a whole pizza and part of a pizza. Show me the photo that is a part of a pizza.3b) If student does not respond or responds incorrectly, examiner removes all stimuli but the photo of part of a pizza and says, [Look at/touch] the photo of part of the pizza.27
Mathematics Item 3a Examiner presents to student an illustration of
two unlabeled pizzas and says, Here are two photos of pizza, a whole pizza and part of a pizza. Show me the photo that is a part of a pizza.
28
Mathematics Item 3b If student does not respond to item 3a or
responds incorrectly, examiner removes the first illustration (of a whole pizza) and shows student only the second illustration. Examiner says, [Look at/touch] the photo of part of a pizza.
29
Author Assessment Tasks
Step 5. Document correct answerItem 1 Item 2 Item 3a/3b
Student indicates the picture of ¾ of a pizza on the worksheet
Student indicates the picture of ½ of a pizza
3a) Student indicates the picture that is not a whole pizza
3b) Student looks at/touches the picture of half of pizza
30
Author Assessment Tasks
Step 6. Describe stimulus materials
31
Item 1 Item 2 Item 3a/3bThree unlabeled drawings of pizzas presented in a row (these are bird’s eye views of the pizzas, not side views with perspective). Each pizza is divided into four slices: one has 2 of the four slices remaining, one has one of the four remaining, and the third has three of the four remaining. Every pizza shows the quarters outlined with a dotted line. Every pizza has four section outlines even if they are missing. Card with ¾ printed on it.
Two unlabeled photos of pizzas presented in a row (these are bird’s eye views of the pizzas, not side views with perspective). One photo is of a whole pizza and the other photo is of a half of a pizza. Card with ½ printed on it.
Two unlabeled photos of pizzas presented in a row (these are bird’s eye views of the pizzas, not side views with perspective). One photo is of a whole pizza, the second photo is of half of a pizza.
Author Assessment Tasks
Step 7. Describe materials for the examiner
32
Item 1 Item 2 Item 3a/3b3 pictures of pizzas 1 card with numeric ¾Recording sheet for teacher to complete
2 photographs of pizzas
1 card with numeric ½
Recording sheet for teacher to complete
2 photographs of pizzas
Recording sheet for teacher to complete
ELA Item 1 Examiner says, This passage is from Roll of Thunder,
Hear My Cry. Examiner reads passage to student:
Cassie tells Mama that a mob is going to come after a boy. Papa runs outside with a gun. Soon, Mama sees that their cotton field is on fire. She thinks it started by lightning. Then Cassie goes back to her room and wonders if people can live in peace.Everyone goes out to the cotton field. Papa and Mama's neighbors are helping them put out the fire. The people in the mob are helping put out the fire too. Cassie thinks that her Papa started the fire.
33
ELA Item 1 Examiner lays out a card with the following
question and reads the question to the student: How did Papa starting the fire change the story? Examiner then lays out a card with each of the following sentences and points to each sentence as it is read.
1. The fire made the mob more angry and they came after the boy
2. The mob helped put out the fire and didn't hurt the boy
3. Cassie got scared of the fire and ran away from home
34
ELA Item 2 Examiner says, This passage is from Roll of
Thunder, Hear My Cry. Examiner reads passage to student:
A mob is going to come after a boy. His friends are warned that the mob is coming. One of the friends sets his own cotton field on fire to lead the mob away. All the neighbors and the mob come together to put out the fire. The boy is not hurt.
35
ELA Item 2 Examiner lays out a card with the following question
and reads the question to the student: What happened next after the friend set the cotton field on fire? Examiner then lays out a card with each of the following sentences and points to each sentence as it is read.
1.All the neighbors and the mob put out the fire2.The boy’s friends are warned that the mob is
coming3.A mob decides to come after the boy
36
ELA Item 3a Examiner places a picture card in front of student
and reads the first sentence from the passage: Cassie goes to the store.
Examiner places the second picture card in front of student and reads the second sentence from the passage: The storekeeper yells at Cassie.
Examiner places the third picture card in front of student and reads the third sentence from the passage: The storekeeper makes Cassie get out of the store.
37
ELA Item 3a Examiner removes all pictures and then presents
student with two pictures (Picture 1 and Picture 3) and says, Look at these pictures. Which picture shows what happened after the storekeeper yelled at Cassie?
ELA Item 3b If student cannot or does not respond, examiner
removes Picture 1 leaving only Picture 3 in front of student and says, [Look at/touch] the picture that shows what happened after the storekeeper yelled at Cassie.
38
Design Patterns Library of Design Patterns and Tasks
AAD-Mathematics and AAD-ELA projects, when completed, will have produced approximately 50 Design Patterns and Task families–200 items
Spans grades 3–8 and high school Math Design Patterns organized around NCTM
Standards and Expectations ELA Design Patterns organized around Common
Core State Standards Supports design of multiple-choice, scenario-
based, and portfolio assessment formats
39
Benefits of Using ECD with UDL
What states gain from applying this approach in assessment design: Content-valid assessment tasks designed to
align to state or national standards Increased efficiency through systematic task
design and development-reusable design patterns and task templates
Built-in documentation of design decisions using PADI (NSF and SRI development funding)
Variation in DOK and complexity Attention to UDL throughout design process
40
Pilot and Refine TasksGOAL: In Assessment Delivery, the newly developed assessment tasks are pilot tested and refined based on empirical resultsAll newly developed assessment tasks must be empirically studied to establish their feasibility, reliability, and validityMethodologies may include cognitive labs, teacher surveys, field tests of tasks, and student observations
41
Pilot and Refine Tasks
Math Task Try-outsResearch Questions: Task and Item Viability
Can the 3 items within the exemplar task be administered as designed?
Is the task clear to the student and to the teacher?
Appropriateness for a Range of Student Performance Levels Can any of the students do the most difficult item? Can most students do the least difficult item?
42
Math Task Try-outs Detailed Task Materials and Instructions Data Collection Booklet: 85 questions
Score for each item administered Item related information
Student received instruction related to the item Item characteristics – directions, graphics,
manipulatives and materials, language, complexity
Student information Communication level, disability, grade-level
Teacher information Years of experience, familiarity with academic
standards43
AAD-M Idaho - One State’s Experience with PADI & ECD
PADI and ECD Pluses!Thorough Stepwise Methodology – for AA ECO Based Item Development; based upon DOK, UDL fits with LALStakeholder Participation – Assessment, Curriculum & Ed SRI – Pioneering Methodology – AA Crosswalk with Multiple State ECOs (precursor to Common Core)Increased # Tools in Educators’ Toolbox for formative classroom assessmentIncreased Educator Awareness of AA and Ties to Instruction
44
Flow Chart
There are two paths: Student responds
correctly to A1 Student responds
incorrectly to A1
45
Administer
Item A1
Administer
Item A2
Stop administration
Stop administration
Correct response
Administer
Item B
Administer
Item C
Stop administration
Correct, incorrect, no response, or refused
Incorrect or no response
Correct, incorrect, no response, or refused
Note: Be sure to gain the student’s attention before presenting EACH item.
Refused
How to Administer Task Items
Sequence of Items within a Task: General Case When Item A1 is Correct
46
Sequence of Items within a Task: Special Case When Item A1 is Incorrect
47
Idaho AAD-M Task Try-out Incentives!
Stipend for Teachers
Assessment and Instructional Materials
Student Rewards Shopping &Budgeting withDouble DigitSubtraction
48
Math Task Try-outs Number of student participants: 192 Number of teacher participants: 55 Number of tasks administered by
state: 16 9 tasks/suites of items common across
all states 7 tasks/suites of items unique to each
state
49
Math Task Try-outs Fidelity of Implementation
Of the 1,547 item administrations:98% (1,514) items were administered as intended
2% (29) items were not administered as intended
50
Math Task Try-outs
51
Math Task Try-outs
52
Math Task Try-outs
53
Percent of Items Correct by Student Communication Level
Task Try-outs Preliminary Results
54
Task Try-outs Preliminary Results
55
Task Try-outs Preliminary Results
56
Idaho Task Tryout VideoVideo Demonstration Illustrates Sequence of Items Within a Task and Ties to Instruction.
57
8th Grade Student Mathematics Number and Operations A1
Idaho – AAD-M Next Steps
Ties to Instruction Outcomes Workshop and Learning Community 68 Educators 2 day hands-on Materials 4 live sites (split screen) Tanberg Video Teleconferencing Interactive Capacity to view:
Demonstration/Lecture Websites Documents (via doc camera) Presenters PowerPoint
58
Idaho AA Learning Community Lesson Plan & Resources
Alternate Assessment Learning Community AA Mathematics Lesson Plan Library
http://itcnew.idahotc.com/dnn/iaa/AADocuments.aspx
59
Task Refinement Individual tasks can be refined on the
basis of item level results from the try-outs: Reword prompts and answer choices Revise item format (e.g., multiple choice to
constructed response) Adjust stimulus materials Refine scoring rubrics Increase or decrease use of supports for
Additional KSAs Recommend changes in modes of
perception and expression used in tasks60
Summary ECD is well-suited to developing alternate
assessment tasks; supports integration of UDL ECD allows the systematic documentation of
assessment tasks to support efficiency of task development (re-usability)
ECD supports the design/development of a range of items with varying levels of complexity and DOK
The co-design approach actualizes the value of the special educators, content specialists and assessment specialists
61
Panel Questions1. Describe the co-design process in your
state.2. What were the benefits/challenges of using
evidence centered design/universal design for learning as the frameworks for task development?
3. How are the tasks being used in your state now?
4. What is the future of this approach in your state/consortium?
62
Contact Us http://alternateassessmentdesign.sri.com
[email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
63