Designing Validity Into an Alternate Assessment
description
Transcript of Designing Validity Into an Alternate Assessment
The present publication was developed under grant 84.373X100001 from the U.S. Department of Education, Office of Special Education Programs. The views expressed herein are solely those of the author(s), and no official endorsement by the U.S. Department should be inferred.
Designing Validity into an Alternate Assessment
Neal Kingston, PhDAngela Broaddus, PhD, Meagan Karvonen, PhD
Center for Educational Testing and Evaluation, University of Kansas
Karen Erickson, PhDCenter for Literacy & Disability Studies,
University of North Carolina at Chapel Hill
Moderator: Susan Weigert, PhD, OSEP
Designing Validity Into an Alternate Assessment
Overview of Evidence Centered Design
3
Five Layers of ECDDomain analysis
Domain modeling
Conceptual assessment framework
Assessment implementation
Assessment delivery
4
Domain Analysis• Descriptions of the knowledge
domain being assessed• Situations or activities that evoke
the relevant concepts and skills• Common misconceptions• Instructional strategies
5
Domain Modeling• Identify focal knowledge and skills• Identify potential tasks for
assessing focal knowledge and skills
• Describe characteristics and variable features of assessment tasks
6
Conceptual Assessment Framework
• Student models– proficiencies the assessment aims to
address• Task models– how students will respond in an
assessment situation to indicate their knowledge
• Evidence models– how observations collected from tasks
offer evidence about what students know
7
Assessment Implementation• Task/testlet development • Scoring rules• Measurement models
8
Assessment Delivery• Tests administered to students• Tests scored• Reports generated and
interpreted
Overview of the DLM Maps, Claims, Conceptual Areas, and Essential
Elements
Learning Map
Claims
Conceptual Areas
Essential Elements
Overview of the DLM Maps, Claims, Conceptual Areas, and Essential
Elements
11
Learning Map
12
Models need to specify claims… (Gong, 2012)
13
DLM ClaimsEnglish Language Arts – Claim 1
Students will comprehend text in increasingly complex ways
Mathematics – Claim 1Students will demonstrate increasingly complex understanding of number sense.
14
Conceptual Areas• Conceptual areas are comprised of
nodes that represent the development of related cognitive processes.
• Conceptual areas contain:– nodes that have been identified as
the targets for Essential Element, and– nodes preceding and extending
beyond the targets.
ELA.C1.1 Determine Critical Elements of Text
15
ELA.C1.1 Determine Critical Elements of Text
16
English Language Arts Claims and Conceptual
Areas
17
English Language Arts Claims
ELA.C1 Students can comprehend text in increasingly complex ways.
ELA.C2 Students can produce writing for a range of purposes and audiences.
ELA.C3 Students can communicate for a range of purposes and audiences.
ELA.C4 Students can engage in research/inquiry to investigate topics and present information.
18
ELA.C1 Students can comprehend text in increasingly complex ways. ELA.C1.1 Determine Critical Elements of
Text ELA.C1.2 Construct Understandings of
Text ELA.C1.3 Integrate Ideas and
Information from Text
19
ELA.C2 Students can produce writing for a range of purposes and audiences.
ELA.C2.1 Use Writing to Communicate
ELA.C2.2 Integrate Ideas and Information in Writing
20
ELA.C3 Students can communicate for a range of purposes and audiences.
ELA.C3.1 Use Language to Communicate with Others
ELA.C3.2 Clarify and Contribute in Discussion
21
ELA.C4 Students can engage in research/inquiry to investigate topics and present information.ELA.C4.1 Use Sources and Information
ELA.C4.2 Collaborate and Present Ideas
Mathematics ClaimsM.C1
Number Sense: Students demonstrate increasingly complex understanding of number sense.
M.C2
Geometry: Students demonstrate increasingly complex spatial reasoning and understanding of geometric principles.
M.C3
Measurement, Data and Analysis: Students demonstrate increasingly complex understanding of measurement, data, and analytic procedures.
M.C4
Algebraic and functional reasoning: Students solve increasingly complex mathematical problems, making productive use of algebra and functions.
Claim M.C1 – Number SenseStudents demonstrate increasingly complex understanding of number sense. M.C1.1 Understand number structures
(counting, place value, fraction) M.C1.2 Compare, compose, and
decompose numbers and sets. M.C1.3 Calculate accurately and
efficiently using simple arithmetic operations
Claim M.C2 - GeometryStudents demonstrate increasingly complex spatial reasoning and understanding of geometric principles. M.C2.1 Understand and use geometric
properties of two- and three-dimensional shapes
M.C2.2 Solve problems involving area, perimeter, and volume.
M.C3 – Measurement, Data, and Analysis
Students demonstrate increasingly complex understanding of measurement, data, and analytic procedures. M.C3.1 Understand and use
measurement principles and units of measure
M.C3.2 Represent and interpret data displays
M.C4 – Algebraic and Functional Reasoning
Students solve increasingly complex mathematical problems, making productive use of algebra and functions.M.C4.1 Use operations and models to
solve problems M.C4.2 Understand patterns and
functional thinking
EE Concept Maps take content to assessment design
• Identify claim, conceptual area, CCSS, and Essential Element
• Identify key vocabulary• Describe and define a range of skill
development• Describe and define misconceptions• Identify prerequisite and requisite skills• Identify sensitivity and bias barriers• Identify accessibility issues
Essential Element Concept Map
28
Feelings of CharactersELA.EE.RL.3.3 Identify the feelings of the characters in a story
Embedded and/or Conclusion
Both Embedded
Both Embedded
FractionsM.EE.3.NF1-3 Differentiate a fractional part from a whole
Dissecting the EECM
Framework• Claim• Conceptual Area• Common Core State Standard• Essential Element• Essential Question– “Does” versus “Can”
• Use “Does the student…?” when asking a comprehension question.
• Use “Can the student…?” when asking a performance question.
– Questions focus on EE, address linkage node
Vocabulary• Concepts • Words
Nodes• Initial (a)• Distal (b)• Proximal (c)• Target (d)• Successor (e)• Supporting Nodes
Questions/ Misconceptions• Questions to Ask–What additional skills does the
student need to reach the next node?
• Misconceptions and Errors–What is preventing the student from
reaching the next node?
Next EE/Accessibility/ Barriers
• Next EE• Accessibility• Barriers• Testlet Access
Instructionally Relevant Testlets
TESTLET DEVELOPMENT
Testlet Development Process
• Follows a rolling sequence in conceptual areas, driven by EECM development
• One item writer develops a set of testlets
Testlet Sets
InitialPrecursor
Target
Successors
Connect the map…
Behavior Testlet a
…to the items developed.
DistalPrecursor
ProximalPrecursor
Behavior
Behavior
Behavior
Behavior
Testlet b
Testlet c
Testlet d
Testlet e
DLM Testlet Development Process
Dynamic Learning Maps
Structure of a Testlet• Begins with engagement activity• ELA: Text presented in segments
with questions embedded and at conclusion
• Math: series of questions or problems related to single topic
SAMPLE TESTLETS
TESTLET ADMINISTRATION
DISCUSSION
Questions1. What features must be in place for
alternate assessments to be instructionally relevant and not just embedded?
2. What factors might support teachers in making full use of an assessment like DLM?
3. What are the important research questions to be asked about instructionally relevant alternate assessments?
THANK YOU!For more information, please go to:
www.dynamiclearningmaps.org
Thank you!