PARCC’s Online Score Report Designs Jeff Nellhaus, Chief of Assessment PARCC, Inc. December 8, 2014.
Plans for Reporting Results and Performance Level Setting The Second Annual New Jersey Leadership...
-
Upload
bruce-johnston -
Category
Documents
-
view
213 -
download
0
Transcript of Plans for Reporting Results and Performance Level Setting The Second Annual New Jersey Leadership...
Plans for Reporting Results and Performance Level Setting
The Second Annual New Jersey Leadership SummitJanuary 21, 2015
Jeffrey NellhausChief of Assessment, PARCC, Inc.
1. PrioritiesProvide a brief review of the priorities that drive PARCC
2. Reports of ResultsYou will become knowledgeable about the kinds of metrics and reports that will be used to report the results of the PARCC assessments next fall
3. Performance Level Setting (aka standard setting)You will be knowledgeable about the process PARCC intends to conduct this summer to identify the threshold scores for each of its five performance levels
Objectives for this morning …
2
PARCC Consortium in 2015
3
AR
COIL
LA
MA
MS
NM
NY
OH
TN
MD
NJ
RI
Governing Board States
DC
• Administering the test in 12 members
• 5 million students
• Nearly 15% of students in grades 3-11 nationally
• About 3 in 4 will take the tests on computer
Develop tests worth taking
Texts worth readingEssays worth writing
Problems worth solvingAccessible to all students
Goals …
Report resultsworth using
ValidReliable Timely
Actionable
4
Determine whether students are college- and career-ready or “on track”
Provide tools to assess student learning and support instruction
during the school year
Report comparable results across schools, districts and member states
Generate data, including growth metrics, for accountability uses
Priority Purposes
Use technology to increase student access and engagement, and
create efficiencies
5
PARCC Assessment System
6
Draft Score Report Designs“Results Worth Using”
7
PARCC Score Report Design Study
• PARCC conducted a 5-month, multi-phase Score Report Design Study to establish priorities for the kinds of data the reports should include and how they should be presented.
• Stakeholders providing input included:Teachers and Administrators (Central Office and Building)ParentsHigher Education RepresentativesPARCC state education leadershipPARCC Technical Advisory Committee (TAC)
8
Data Management & Reporting System
• PARCC score reports will be available to local educators online through its interactive web-based Data Management and Reporting System, and to parents via a printed student reports
• PARCC’ web-based system will provide authorized users to customize reports using the following “filters”
9
StudentSchool DistrictState PARCC
YearSubject
Type of TestGrade
Student Group
Overall ScoresStatus Scores
Growth ScoresSub-Scores
Item Level Scores
Performance Levels
•Math•ELA/L
Scale Scores
• Mathematics• ELA/L
• Reading• Writing
Sub-Scores
•Whether the student’s performance in the sub-category was At/Above, Near, or Below the performance of students who scored “just Level 4” overall in the content area
Metrics for Reporting Results
• At the level, student the number of points earned
• At the school level, the average number of points earned by all the students in the school who took the item.
Item Scores
Draft of Individual Student Report
11
Draft of Student Roster - Sub-Scores
12
Draft of Student Roster – Item Scores
13
Draft School by Grade Summary Report
14
Draft District by School Summary Report
15
Draft State by District Summary Report
16
Draft PARCC by State Summary Report
17
Performance Level Setting
18
• PARCC will report results using five performance levels (names under discussion)
• Students performing at Levels 4 or 5 on Algebra II, Integrated Math 3, or ELA/L grade 11 assessments will earn a College- and Career-Ready Determination
• Public colleges and universities in PARCC states have agreed
to allow students who have been accepted and have earned CCR Determinations to enter directly into credit-bearing courses without remediation
PARCC Performance Levels
19
Performance Level Setting: What is it?
20
Use Performance Level Descriptors & Data from Research
Identify Threshold Scores
Five PARCC Performance Levels
Who will participate?
• High School Panels (6)• Grade 3-4 Panel (2)• Grade 5-6 Panel (2) • Grade 7-8 Panel (2)
Performance Level Setting
21
K-12 educators
Postsecondary faculty
Business people
Grade-Span Panels
Each Panel will Include 20
Participants
Schedule of Performance Level Setting Events
22
PARCC Performance Level Setting Schedule
Panel Start Date End Date
Subject/Grade
Mathematics PanelsELA/Literacy
Panels
Dry Run 5/21/2015 5/28/2015 Algebra II and Integrated 3 Grade 11
High School 7/27/2015 7/31/2015
Algebra I and Integrated 1
Geometry and Integrated 2
Algebra II and Integrated 3
Grade 9
Grade 10
Grade 11
Grades 7–8 8/17/2015 8/21/2015 Grades 7 and 8 Grades 7 and 8
Grades 3–6 8/24/2015 8/28/2015Grades 5 and 6
Grades 3 and 4
Grades 5 and 6
Grades 3 and 4
Performance Level Setting:What Process Will be Used?
PARCC will Use an Evidence-Based Process• The process combines expert judgment with data from
research
• The process will generate threshold scores that – Reflect PARCC’s performance level descriptors (PLDs)– Are consistent with the results reported by other well-regarded
state, national and international assessment programs
• Because comparability is a priority, every state will adopt the same threshold scores
23
Performance Level Setting: Training
Panelists will:• Practice with online tutorial • Take the computer-based assessment• Review the performance level descriptors (PLDs)• Write “borderline” performance level descriptors
(PLDs)• Practice with the process
24
Proposed Performance Level Setting Method
25
Yes/No Method (proposed)
Review test items
• Answer the question:
• Based on the PLDs, would a student performing at Level X be likely to answer the item correctly?
Impact data
• Impact data includes:• Empirical
data from research
• Judgments of other panelists
• % of students performing at each level
• % who got item correct
Grade 3 Sample Item and PLDs
Grade 3 Math : Sub-Claim AThe student solves problems involving the Major Content for the grade/course with connections to the Standards for
Mathematical Practice.Level 5: Distinguished
Command Level 4: Strong Command Level 3: Moderate Command Level 2: Partial Command
Time 3.MD.1-13.MD.1-2
Tells, writes and measures time to the nearest minute. Creates two-step real world problems involving addition and subtraction of time intervals in minutes.
Tells, writes and measures time to the nearest minute. Solves two–step word problems involving addition and subtraction of time intervals in minutes.
Tells, writes and measures time to the nearest minute. Solves one-step word problems involving addition or subtraction of time intervals in minutes.
Tells, writes and measures time to the nearest minute. Solves one-step word problems involving addition or subtraction of time intervals in minutes, with scaffolding, such as a number line diagram.
26
Patricia needs to read for 120 minutes each week.• She read for 26 minutes on Monday.• She read for 39 minutes on Tuesday.• She read for 38 minutes on Thursday.
How many more minutes does Patricia need to read this week?
minutes
Click on the answer box and type in a number. Backspace to erase.
Research to Inform the Performance Level Setting Process
• Research Questions– What percentage of students are likely to be college- and career-ready,
according to PARCC’s definition?– What percentage of students are likely to be on track to college and career
readiness at earlier grades?
• Two studies to inform performance level setting
27
Postsecondary Educators
Judgment Study
Benchmark StudySATACT
NAEPTIMSSPISA
Performance Level Setting Timeline
28
Sign up for the PARCC Updates
newsletter at www.parcconline.org
• E-mail me [email protected]
• Follow me on Twitter• @jmnellhaus
• Follow us on Twitter @parccplace
• Visit our website www.parcconline.org
• http://parcc.pearson.com/
• Sign up for the PARCC Updates newsletter on our website
Updates and more information
29