Honors Project Christina Olson Spring 2014 EDU 221 Dr. Butler.
John F. Olson, Ph.D. Principal Investigator & Project Coordinator, CTEAG Project
-
Upload
winter-owens -
Category
Documents
-
view
45 -
download
2
description
Transcript of John F. Olson, Ph.D. Principal Investigator & Project Coordinator, CTEAG Project
Establishing the Validity of Test Accommodations for Students with
Disabilities: A Collaborative of State-based
Research
CTEAG Project Summary of Accomplishments and
Dissemination of Final ProductsJohn F. Olson, Ph.D.
Principal Investigator & Project Coordinator,
CTEAG Project
CCSSO NCSA and Joint ASES-TILSA MeetingDetroit, MI
June 23, 2010
CTEAG Project - OverviewGoals • Conduct research that focuses on validity of
test accommodations and interpretations of scores from accommodated state assessments
• Use a common research design to conduct the studies and to analyze data for comparisons across states and accommodations
• Build a shared body of research-based validity evidence on test accommodations across multiple states
• Share findings from project that provide states with additional evidence (e.g., to submit to the USED for peer reviews) in support of their decisions on the use of various accommodations
2
CT EAG Project Structure• Lead State – Connecticut
• Other Participating States – Kentucky, Michigan, and Nevada
• Additional State Input and Involvement – from ASES and TILSA members
• Project Management Team – group of SCASS advisors, experts, and consultants that oversaw and managed all validity studies and reporting activities
• CCSSO -- provided logistical support, staffing, and consultants to do tasks for the project
CTEAG Research Questions1. Does the accommodation affect item
and test form statistics?
2. Does the accommodation change the content structure of the test, i.e., is there an impact on the test structure under accommodated and standard administrations?
3. Does the accommodation increase performance of students? Does it increase performance more for SWD than it does for SWOD?
CT EAG – Validity of Accommodations Project
State Participation – Details of Studies
5
Topics/Issues CT #1 CT #2 KY MI NVAccommodation Computerized
text reader (computer voice)
Computerized text reader with human voice and other mods made since CT#1
Read aloud Enhanced directions and fewer response options
Read aloud by a trained individual (e.g teacher or aide)
Grade 7 5 4 6 7
Content Area Reading comprehension
Math Reading ELA Math
Administration:Test DateStudy DesignItems/Forms development
May 2008.Computer-based test forms with 20 MC items per form.
Spring 2009.Computer-based test forms.Used old secure math test with 26 MC items per parallel form.
Mar 2009.Developed forms from a pool of non-state items that were similar in content to KY forms with 22MC items per form.
Oct 2008 and Jan. 2009.Used released items – 24 MC and 1CR per form.
Nov-Dec 2008. Used 20 MC items per form.
Summary of Results
John Olson Establishing the Validity of Test Accommodations June 2009
6
• In most cases, accommodations did not alter item stats (difficulty, discrimination)
• Accommodations also did not change test form stats/characteristics (mean item difficulty, reliability)
• Accommodations did not have an effect on content structure of test forms
• Accommodations did not change dimensionality of tests
Summary of Results (continued)
John Olson Establishing the Validity of Test Accommodations June 2009
7
• Overall, results from ANOVA were mixed• Accommodations increased performance
of students in KY study; enhanced directions increased performance in MI
• Differential boost effect found in some cases, i.e., SWD scores increased more than SWOD in CT#2, KY, and MI studies
• Key areas of focus – • Differential boost effects • Increasing the accessibility of the
assessment
CTEAG Project - Accomplishments
• Assisted and guided states in conducting 5 different validity studies on test accommodations
• Analyzed all data from the studies and reviewed results with the PMT and states (and sometimes with states’ vendors)
• Summarized results in a series of detailed state reports
• Shared updates and information on progress of project at 8 ASES and TILSA meetings from 2007 to 2010
• Completed and disseminated 3 reports and a Project Database
• Presented information on the project at 4 national conferences and meetings (2007-2010)
• Conducted 3 evaluations of the work, surveying states, and summarizing the info in an Evaluation Report
C
8
CTEAG Final Reports/Products• Guidebook for Test Accommodations
Validity Studies --provides overall summary of results/findings and guidance to states on validity studies in a nontechnical manner [report was distributed to ASES and TILSA in Feb. 2010]
• CTEAG Technical Report – provides technical information on the project and details of each state’s study, e.g., designs, methods, analyses, statistics, results, etc.
9
CTEAG Final Reports/Products• Lessons Learned from State Validity
Studies – provides information on the background and context in which the validity studies were conducted, issues that states may need to consider, and lessons learned by the four participating states
• CTEAG Project Database -- provides additional info from the five studies, such as, samples, variables, data files, and statistical data for use in further research
10
Dissemination Activities• Presentation Session at 2010 NCSA on
June 22−State participants present results and
perspectives from conducting validity studies−Representative from USED discusses
importance of findings from federal perspective
• Dissemination Workshop at Joint ASES-TILSA meeting on June 23−Share all remaining products and summary of
accomplishments−State participants discuss lessons learned and
how data and findings can be used to benefit states
−Expert advisors and USED representative discuss implications at national level
11
Final Dissemination Activities• Distribute the reports/products widely to
states, the USED, and others
• Place PDFs of all final products on websites (CSDE , CCSSO/EAG)
• Issue formal announcement (press release?) and Chiefline newsletter article
• Coordinate with project partners (NCEO and NASDSE) to share information with others
• Submit final report on project to USED12
Questions &
CommentsFor more information, contact:
John F. OlsonPresident/Founder
Olson Educational Measurement & Assessment Services
617-965-1490 (home/office)
210-859-5593 (cell)[email protected]