State Special Education Outcomes: Results of a 2009 Survey of States
description
Transcript of State Special Education Outcomes: Results of a 2009 Survey of States
1
State Special Education Outcomes: Results of a 2009 Survey of States
Jason Altman – NCEOMari Quenemoen – NAAC
TASH Annual Conference – Nov. 19, 2009
2
Introduction
• It is important that state large-scale assessment systems be both inclusive of and fair to all populations including students with disabilities
• A 2009 survey explored these issues as a whole and paid special attention to state practices in assessing students who may participate in an alternate assessment option
3
Survey
• All 50 states returned the 12th survey of state directors of special education and state directors of assessment by the National Center on Educational Outcomes (NCEO)
• The survey was disseminated to respondents electronically in the winter of 2009
• NCEO returned surveys for verification and, upon receipt, entered, reviewed, and analyzed the data
• Results of the eleventh survey were presented at the 2008 TASH annual conference
4
Presentation Purpose
• The purpose of this presentation is to provide a snapshot of new initiatives, trends, accomplishments, and emerging issues in the large scale assessment of students with disabilities
• Presenters will pay special attention to issues surrounding the testing of students with disabilities who may be assessed using an alternate assessment
Apples to Oranges or Apples to Apples
6
General Issues
• States reported that assessment validity and test design/content were areas of much success
• Issues related to English language learners with disabilities and the performance of urban schools were often selected as challenging
• States appear to have a very mixed viewpoint on the AA-MAS (7 respondents in each category)
• On the other hand, respondents appear to be strongly entrenched on the successful side of the ledger as related to reporting and monitoring
7
Regular Assessment IssuesAccommodations
• More than four in five states reported monitoring accommodations use in 2009
Directly observing test administrations
Interviewing students, teachers, and administrators
Conducting desk audits
Sending teams into schools on a scheduled basis
Sending teams into schools on a random basis
Sending teams into schools on a targeted basis
Completing online record reviews
Other
We do not monitor accommodations use
0 5 10 15 20 25
22
15
14
13
10
10
6
10
9
7
2
1
0
3
0
0
1
1
Unique States Regular States
Number of States
8
Regular Assessment IssuesAccommodations
• Most respondents also reported that accommodations cause some difficulty on test day in schools and districts within their state (n = 41)
Arranging for trained readers/interpreters
Ensuring proctors give correct accommodations
Scheduling substitute providers of accommodations
Training proctors in providing accommodations
Ensuring the ordering of special test editions
Ensuring that accommodations are recorded
Arranging for and checking equipment
Ensuring accommodations for makeup assessments
Other
We have ensured accommodations are carried out
0 2 4 6 8 10 12 14 16 18 20
19
17
14
12
11
6
5
1
9
9
3
4
3
3
3
2
4
1
3
0
Unique States Regular States
Number of States
9
Alternate Assessments Based on Modified Academic Achievement Standards
• In April 2007, new No Child Left Behind regulations on AA-MAS were finalized
• States have the option of developing alternate assessments based on modified achievement standards (AA-MAS)
• At the time of the 2007 survey of states we found that five states already had an assessment in place, 32 states were considering modifying an existing grade-level assessment and another twenty-five were considering developing a new assessment
• In the past two years, some states have further refined their motivations for moving in this assessment direction, while other states were instead making efforts to improve the assessments that they already offer
10
Alternate Assessments Based on Modified Academic Achievement Standards
• Since 2007, 14 states have decided not to develop of AA-MAS, while 12 states were still in the process of making their decision
• Of the 24 that have developed, or were developing, an AA-MAS, nine have already given the assessment and analyzed the data, one for the first time in 2008-09
8
1
15
12
14 We have given the as-sessment, and analyzed dataWe will give this as-sessment for the first time in 2008-09We are currently in the process of developing an AA-MASWe are currently research-ing the need for an AA-MASWe have decided not to develop at this time
11
Alternate Assessments Based on Modified Academic Achievement Standards
• States that were developing or have developed their AA-MAS were three times more likely to modify an existing grade-level test rather than design an entirely new test
5
15
2
26
2
Design an entirely new testModify an existing grade-level testOther, please explain belowResearching the Need or Not De-velopingNo Response
12
Alternate Assessments Based on Modified Academic Achievement Standards
• Respondents have been asked on the past two surveys about the changes they envisioned making when modifying existing tests
• Findings were similar to those found in 2007 save for a less frequent response to use of non-traditional items (keeping in mind that many more states have decided not to develop the AA-MAS at this time
Simplify vocabulary
Reduce the number of total test items or time
Use shortened or fewer reading passages
Use fewer answer choices
Include only multiple choice questions
Use non-traditional items or formats
0 5 10 15 20 25 30 35
16
14
13
12
6
3
26
26
29
23
11
14
2007 2009Number of States
13
Alternate Assessments Based on Modified Academic Achievement Standards
• States reported a variety of strategies and methods in determining content targets and blue prints for their AA-MAS
Same specifications as for regular assessment
Stakeholder panels
Review of literature on teaching at-risk learners
Special education curriculum literature
Teacher survey
Test company provided
Consultant developed
Other, please explain below
Researching the Need or Not Developing
No Response
0 5 10 15 20 25 30
17
11
7
6
3
3
1
2
26
2
Number of States
14
Alternate Assessments Based on Alternate Achievement Standards
• Seven states (HI, ID, MS, NE, NH, NV, and UT) and three unique states (Guam, PR, and RMI) are currently in the process of revising their AA-AAS
• Trend analysis from 2005 and 2003 data
15
AA-AAS Topics
• Test formats• Content alignment• Scoring methods• Rubric analysis• Methods for determining achievement levels• Scorers
16
AA-AAS Formats•40% of states use a portfolio or body of evidence (40% of these are standardized)•36% use a standardized set of performance events/tasks (44% of which require the submission of evidence)•16% use a multiple choice test
17
AA-AAS Content Alignment• Complete transition to alignment with academics
• Extended/expanded academic content standards (27 states) or grade level academic content standards (16)
• No states align AA-AAS to functional skills, and IEP teams no longer determine the content of the test for individual students
18
AA-AAS Scoring Procedures
Rubric
Points Assigned on a Rating Scale
Number of Items Correct
Reading Rate or Accuracy
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
80%
16%
10%
4%
74%
32%
24%
8%
66%
16%
38%
4%
2009 2005 2003
Percentage of States
19
Outcomes Measured by Rubrics• Primarily skill/competence, level of assistance, and alignment to academic content
• Trend away from scoring other factors
20
Outcomes Measured by Rubrics (2005 and 2009)
Level of Assistance
Skill/Competence
Alignment with Academic Content Standards
Ability to Generalize
Number/Variety of Settings
Degree of Progress
Appropriateness
Participation in General Education Settings
Support
Staff Support
Self Determination
Social Relationships
Parent Satisfaction
0 5 10 15 20 25 30
2009 2005
Number of States
21
Outcomes Measured by Rubrics ( States and Unique States 2009)
Level of Assistance
Skill/Competence
Alignment with Academic Content Standards
Ability to Generalize
Number/Variety of Settings
Degree of Progress
Appropriateness
Participation in General Education Settings
Support
Staff Support
Self Determination
Social Relationships
Parent Satisfaction
0 5 10 15 20 25 3024
25
15
10
10
6
8
5
6
5
4
3
1
3
5
4
2
1
2
2
1
Unique States States
Number of States or Unique States
22
Methods for Determining Achievement Levels
Contrasting groups
Body of work
Judgmental policy capturing
Reasoned judgment
Bookmarking
Modified Angoff
Other
0% 10% 20% 30% 40% 50% 60% 70%
12%
8%
4%
58%
23%
20%
37%
17%
29%
12%
10%
2009 2005
Percentage of States
23
Who Scores the AA-AAS?
test company contractor
student's special education teacher
teachers from other districts
state education agency
teachers in the same district as student
members of student's IEP team
university contractor
other
currently in revision
0 5 10 15 20 25
21
12
10
4
4
4
3
3
7
1
1
1
1
1
1
3
Unique States States
Number of States or Unique States
24
Thank you! For more information…
National Center on Educational OutcomesUniversity of Minnesota612-626-1530http://www.nceo.info
Jason Altman:[email protected]
National Alternate Assessment CenterUniversity of Kentucky859-257-7672 http://www.naacpartners.org
Mari Quenemoen: [email protected]