Illustration of a Validity Argument for Two Alternate Assessment Approaches
-
Upload
robin-payne -
Category
Documents
-
view
14 -
download
0
description
Transcript of Illustration of a Validity Argument for Two Alternate Assessment Approaches
Illustration of a Validity Argument for Two Alternate
Assessment Approaches
Presentation at the OSEP Project Directors’ Conference
Steve FerraraAmerican Institutes for Research
August 1, 2006
2
Goal
Illustrate planning for the validation process for large-scale assessments using standards-based alternate assessments from two states
Use selected examples from the paper Standards and Assessment Approaches for Students with Disabilities Using a Validity Argument
3
Over-arching concept
Provide evidence to support intended inferences about students
Consideration of assessment design (i.e., tasks, administration conditions, scoring)
Plans for collecting procedural and empirical evidence
Important specific principles and recommendations in the paper
4
Examples from the paper
Massachusetts alternate assessment portfolio (Weiner, 2002)
Oregon performance tasks (Tindal et al., 2003) Mathematics, grades 3-5 Intended inferences
The assessment adequately reflects the domain of knowledge and skills for the construct
The assessment accurately identifies students’ level of proficiency in mathematics
5
Procedural evidence
Test design and development process Quality of the items and tasks Assemblage of items/tasks/evidence into an
assessment Administration and scoring process
6
Empirical evidence
Alignment between the alternate content standards and the assessment items/tasks/evidence (and linkage to grade level/band standards)
Item/task functioning Reliability of scoring and test score interpretations Internal relations among items and tasks Response processes External relations with other measures
7
Target math standards
Massachusetts standards Grades three and four standards that focus on
number sense (seven objectives) and operations (three objectives)
Oregon mathematics standards Numbers, Computation and Operations—Grades
four to five
8
MA: Possible assessment strategies and portfolio products
Addressing access skill(s) (skills embedded in academic instruction)
Alice participates in this activity by assembling money envelopes paired with pictures. Alice works with a classmate who counts the money needed for each item and helps Alice place the correct amount into its corresponding envelope. Alice exchanges these envelopes when making a purchase.
9
Possible portfolio products (cont.)
Teacher note describing the work accomplished by Alice and her classmate
Data collected on Alice’s ability to assemble money envelopes and exchange correct envelopes when making a purchase
Videotape of Alice making a purchase Alice’s choice of money envelopes selected
for her portfolio
10
Oregon item
Standard: Read, write, order, model, and compare whole numbers up to 1,000,000, common fractions, and decimals up to hundredths.
Practice Item 24: Find the missing number in the pattern. 2.6 5.2 ___ 20.8 (A) 7.8 (B) 10.4 (C) 13.0 (D) 15.6
Alternate Assessment Task 11: Order Numbers Present the number cards in this order: 3, 1, 8, 6. Say: Place these numbers in order from smallest to largest.
11
Summary
(MA) Assemble money envelopes with a classmate, make purchases Teacher observations of Alice working, videotape
of making purchases (OR) Order numbers from smallest to largest
12
Test development process
Massachusetts Oregon
Does the actual evidence described in the possible assessment strategy fully reflect this construct?
Have the tasks been adequately developed and assembled into an alternate assessment?
13
Test administration and scoring
Massachusetts Oregon
How well conducted are the test administration and scoring procedures?
Are teachers sufficiently trained in administering and scoring the tests (especially because responses may be scored as partially correct and not just correct or incorrect)?
Does the student independently complete work or is the teacher part of this process? If so, to what extent?
14
Alignment and construct representation
Massachusetts Oregon
Are enough pieces of evidence present to represent the domain and avoid construct under-representation?
Are enough tasks present to represent the domain and avoid construct under-representation?
How closely is the alternate assessment aligned to the state content standards in categorical concurrence, depth of knowledge, range of knowledge, and balance of representation?
15
Rater accuracy and score reliability
Massachusetts Oregon
How accurate is the scoring by trained professional scorers?
How accurate is the scoring by trained teachers?
How consistently and accurately do scores categorize students into performance categories?
Which facets of the assessment process influence scores most (e.g., tasks, raters, administration conditions, occasions)?
16
Conclusion
The same types of validity questions apply for all (alternate) assessment approaches How the questions are posed and the evidence
relevant to those questions may differ Intended inferences, corresponding validity
questions, and evidence: Identify during the conceptualization, design, and
development process Pursue during development and as part of
implementation
17
References
Tindal, G., McDonald, Tedesco, M., Glasgow, A., & Almond, P., Crawford, L., Hollenbeck, K. (2003). Alternate assessments in reading and math: Development and validation for students with significant disabilities. Exceptional Children, 69(4), 481–494.
Wiener, D. (2002). Massachusetts: One state's approach to setting performance levels on the alternate assessment. (Synthesis Report 48). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved Dec. 8, 2005 from http://education.umn.edu/NCEO/OnlinePubs/Synthesis48.html.