Post on 03-Jan-2016
Applying SGP to the STAR Assessments
Daniel BoltDept of Educational Psychology
University of Wisconsin, Madison
Some Unique Features of STAR Assessments
• Same CAT assessment is administered within and across years (possible to generate look-up tables for the calculation of SGP)
• Multiple STAR administrations to a student are possible throughout the year, and can occur at different times and/or frequencies for different students
Why SGP may be Useful with STAR Assessments
• Different STAR scores may be associated with different amounts of measurement error (e.g., extremely low or high STAR scores are sometimes of questionable validity)
• There often exists more/less variability in growth observed across students at different initial STAR score levels
Some Practical Issues Related to Administration of STAR Assessments
• How frequently and at what intervals should STAR be administered to get reliable estimates of end-of-year scores? Does the answer depend on the initial (fall) score of a student?
• Is there practical value in the use of SGP for answering this question (in contrast to alternative approaches, such as ordinary least squares---OLS--- regression methods)?
Goals of the Present Study
• Examine SGP as a methodology for quantifying growth and for studying the precision of end-of-year growth predictions using STAR Assessments
• Compare SGP against competing methodologies (OLS regression) in terms of their reported precision of end-of-year predictions
Grade Math Reading Early Literacy
1 99283 100000 968772 100000 100000 286573 100000 100000 57584 100000 100000 554
5 100000 1000006 100000 1000007 86975 1000008 72599 1000009 20835 61919
10 13810 3745511 9796 2627812 5155 12763
Sample Sizes (Students with Fall, Winter & Spring Assessments)
Evaluating the Precision ofSpring Score Predictions
• How well do winter assessments improve our predictions of end-of-year outcomes?• Are the winter assessments more/less useful depending on the fall score obtained by the student?• In answering these questions, we find it useful to examine changes in the confidence intervals for spring scores defined by the SGP percentile cuts
Academic Year
ST
AR
Ma
th S
cale
d S
core
*
95 %ile
80 %ile
50 %ile
20 %ile
5 %ile
Fall Spring
500
600
700
800
900
1000
Example of SGP Percentile Cuts, One Covariate
Academic Year
ST
AR
Ma
th S
cale
d S
core
**
95 %ile
80 %ile
50 %ile
20 %ile
5 %ile
Fall Winter Spring
50
06
00
70
08
00
90
01
00
0
Example of SGP Percentile Cuts, Two Covariates
Comparison of SGP & OLS Intervals, Math Grade 1
According to OLS Standard Error of Prediction
According to SGP %ile cuts
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
0 200 600
150
200
250
300
350
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
0 400 800 1200
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 5
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 6
STAR Math, Grades 1-6
OLSSGP
0 400 800 1200
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 7
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 8
200 600 1000
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 9
0 400 800
200
300
400
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 10
200 600 1200
200
300
400
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 11
200 600 1000
200
300
400
500
600
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 12
STAR Math, Grades 7-12
OLSSGP
STAR Reading, Grades 1-6
OLS SGP
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
0 400 800
150
250
350
450
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
0 400 800
200
400
600
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
0 400 800
200
250
300
350
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
0 400 800 1400
250
300
350
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 5
0 400 800 1400
200
250
300
350
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 6
0 400 800 1400
100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 7
200 600 1200
0100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 8
0 400 800 1400
-100
0100
300
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 9
0 400 800 1400
-200
0100
300
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 10
200 600 1200
0100
300
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 11
200 600 1200
100
200
300
400
500
Fall Score
Siz
e o
f 80%
Conf. Inte
rval
Grade 12
STAR Reading, Grades 7-12
OLS
SGP
300 500 700 900
200
250
300
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade K
300 500 700 900
100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade 1
300 500 700 900
100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade 2
400 600 800
0100
200
300
400
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade 3
400 600 800
50
150
250
Fall Score
Siz
e o
f 80%
Conf. Inte
rval Grade 4
STAR Early Literacy, Grades K-4
OLSSGP
Adding Winter Scores as Covariates with SGP
• Using 80% interval width curves as a baseline, we can further examine how much the intervals are reduced when adding a winter assessment
• The decline in the 80% interval can be used as an indicator of the added precision provided by the winter assessment
STAR Math Example
STAR Reading Example
STAR Early Literacy Example
Some Examples of STAR Score Patterns
Subject Fall Winter Spring Change in 80% Interval
MATH 501 32 531 (+432)625 91 790 (+321)
3 833 852 (-208)2 846 771 (-209)
READING 392 891 589 (+41)376 915 551 (+40)984 1346 1333 (-148)928 1345 1255 (-256)
LITERACY 835 315 832 (+297)777 312 840 (+282)328 858 823 (-242)306 829 851 (-256)
0 200 400 600 800 1000
02
04
06
08
01
00
Fall Score
De
clin
e in
Siz
e o
f 80
% C
on
f. In
terv
al
Grade 5
STAR Math Example
OLS
SGP
0 400 800
20
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
0 200 60020
40
60
80
100
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
0 400 800
20
60
100
140
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
0 400 800 1200
20
40
60
80
100
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
0 400 800
20
40
60
80
100
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 5
0 400 800
20
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 6
STAR Math, Grade 1-6
OLS SGP
0 400 800 1200
20
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 7
0 400 80020
40
60
80
100
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 8
200 600 1000
20
40
60
80
100
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 9
0 400 800
50
100
150
200
250
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 10
200 600 1200
50
100
150
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 11
200 600 1000
050
100
150
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 12
STAR Math, Grades 7-12
OLS SGP
STAR Reading Example
OLS
SGP
0 400 800
30
40
50
60
70
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
0 400 80030
40
50
60
70
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
0 400 800
20
40
60
80
100
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
0 400 800
25
30
35
40
45
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
0 400 800 1400
30
35
40
45
50
55
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 5
0 400 800 1400
35
40
45
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 6
STAR Reading, Grades 1-6
OLS SGP
0 400 800 1400
36
38
40
42
44
46
48
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 7
200 600 120020
25
30
35
40
45
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 8
0 400 800 1400
10
20
30
40
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 9
0 400 800 1400
-20
020
40
60
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 10
200 600 1200
10
20
30
40
50
60
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 11
200 600 1200
010
20
30
40
50
60
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 12
STAR Reading, Grades 7-12
OLS SGP
300 400 500 600 700 800 900
02
04
06
08
01
00
12
0
Fall Score
De
clin
e in
Siz
e o
f 8
0%
Co
nf. In
terv
alGrade 1
STAR Early Literacy Example
OLS
SGP
300 500 700 900
20
30
40
50
60
70
80
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade K
300 500 700 900
020
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 1
300 500 700 900
020
40
60
80
120
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 2
400 600 800
020
40
60
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 3
400 600 800
-10
010
20
30
40
50
Fall Score
Declin
e in
Siz
e o
f 80%
Conf. Inte
rval
Grade 4
STAR Early Literacy, Grades K-4
OLS SGP
Evaluating Predicted Spring Scores in Terms of State Proficiency Thresholds
• The accuracy of SGP and OLS predictions can also be compared against the thresholds associated with state-specific proficiency categories
• By assuming normally distributed residuals (with constant variance) for OLS, SGP and OLS can each be used to define a probability that the spring score will exceed a predefined threshold
Academic Year
ST
AR
Ma
th S
ca
led
Sco
re
**
95 %ile
80 %ile
50 %ile
20 %ile
5 %ile
Fall Winter Spring
50
06
00
70
08
00
90
01
00
0
ProficiencyThreshold
Example of SGP Percentile Cuts against State-Defined Proficiency Category
MATH READING
GradeProficiency
Cuta Percentile OLS R2,b SGP R2,bProficiency
Cuta Percentile OLS R2,b SGP R2,b
3 617 48 .40 .41 445 48 .53 .66
4 709 61 .40 .40 531 52 .53 .68
5 758 60 .42 .42 592 49 .52 .69
6 810 65 .46 .46 647 42 .52 .70
7 823 74 .48 .49 825 51 .55 .74
8 859 68 .47 .48 933 53 .55 .73
aEstimated STAR cutscores for the Tennessee Comprehensive Assessment Program (TCAP)bEfron’s Pseudo R2
Comparing SGP and OLS on Accuracy of Proficiency Predictions
Extending SGP to Accommodate Multiple Intermediate Assessments
• How can additional intermediate assessments be used in SGP to further improve predicted spring scores?•Challenge: Handling varying-time point assessment schedules•One possible solution: Linear interpolation to fixed node locations
Conclusions and Future Directions
• Our SGP analyses suggest substantial variability in the precision of spring score predictions for STAR Math, Reading and Early Literacy depending on fall scores
• There is clear value in incorporating winter assessments into SGP---the largest value occurs for students with extreme fall scores in STAR Math, intermediate fall scores in STAR Reading
• More experimentation needed to determine how best to make use of multiple intermediate assessments within SGP