Applying SGP to the STAR Assessments

Daniel BoltDept of Educational Psychology

University of Wisconsin, Madison

Some Unique Features of STAR Assessments

• Same CAT assessment is administered within and across years (possible to generate look-up tables for the calculation of SGP)

• Multiple STAR administrations to a student are possible throughout the year, and can occur at different times and/or frequencies for different students

Why SGP may be Useful with STAR Assessments

• Different STAR scores may be associated with different amounts of measurement error (e.g., extremely low or high STAR scores are sometimes of questionable validity)

• There often exists more/less variability in growth observed across students at different initial STAR score levels

Some Practical Issues Related to Administration of STAR Assessments

• How frequently and at what intervals should STAR be administered to get reliable estimates of end-of-year scores? Does the answer depend on the initial (fall) score of a student?

• Is there practical value in the use of SGP for answering this question (in contrast to alternative approaches, such as ordinary least squares---OLS--- regression methods)?

Goals of the Present Study

• Examine SGP as a methodology for quantifying growth and for studying the precision of end-of-year growth predictions using STAR Assessments

• Compare SGP against competing methodologies (OLS regression) in terms of their reported precision of end-of-year predictions

Grade Math Reading Early Literacy

1 99283 100000 968772 100000 100000 286573 100000 100000 57584 100000 100000 554

5 100000 1000006 100000 1000007 86975 1000008 72599 1000009 20835 61919

10 13810 3745511 9796 2627812 5155 12763

Sample Sizes (Students with Fall, Winter & Spring Assessments)

Evaluating the Precision ofSpring Score Predictions

• How well do winter assessments improve our predictions of end-of-year outcomes?• Are the winter assessments more/less useful depending on the fall score obtained by the student?• In answering these questions, we find it useful to examine changes in the confidence intervals for spring scores defined by the SGP percentile cuts

Academic Year

ST

AR

Ma

th S

cale

d S

core

*

95 %ile

80 %ile

50 %ile

20 %ile

5 %ile

Fall Spring

500

600

700

800

900

1000

Example of SGP Percentile Cuts, One Covariate

Academic Year

ST

AR

Ma

th S

cale

d S

core

**

95 %ile

80 %ile

50 %ile

20 %ile

5 %ile

Fall Winter Spring

50

06

00

70

08

00

90

01

00

0

Example of SGP Percentile Cuts, Two Covariates

Comparison of SGP & OLS Intervals, Math Grade 1

According to OLS Standard Error of Prediction

According to SGP %ile cuts

0 400 800

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 1

0 200 600

150

200

250

300

350

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 2

0 400 800

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 3

0 400 800 1200

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 4

0 400 800

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 5

0 400 800

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 6

STAR Math, Grades 1-6

OLSSGP

0 400 800 1200

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 7

0 400 800

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 8

200 600 1000

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 9

0 400 800

200

300

400

500

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 10

200 600 1200

200

300

400

500

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 11

200 600 1000

200

300

400

500

600

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 12

STAR Math, Grades 7-12

OLSSGP

STAR Reading, Grades 1-6

OLS SGP

0 400 800

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 1

0 400 800

150

250

350

450

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 2

0 400 800

200

400

600

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 3

0 400 800

200

250

300

350

400

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 4

0 400 800 1400

250

300

350

400

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 5

0 400 800 1400

200

250

300

350

400

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 6

0 400 800 1400

100

200

300

400

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 7

200 600 1200

0100

200

300

400

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 8

0 400 800 1400

-100

0100

300

500

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 9

0 400 800 1400

-200

0100

300

500

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 10

200 600 1200

0100

300

500

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 11

200 600 1200

100

200

300

400

500

Fall Score

Siz

e o

f 80%

Conf. Inte

rval

Grade 12

STAR Reading, Grades 7-12

OLS

SGP

300 500 700 900

200

250

300

Fall Score

Siz

e o

f 80%

Conf. Inte

rval Grade K

300 500 700 900

100

200

300

400

Fall Score

Siz

e o

f 80%

Conf. Inte

rval Grade 1

300 500 700 900

100

200

300

400

Fall Score

Siz

e o

f 80%

Conf. Inte

rval Grade 2

400 600 800

0100

200

300

400

Fall Score

Siz

e o

f 80%

Conf. Inte

rval Grade 3

400 600 800

50

150

250

Fall Score

Siz

e o

f 80%

Conf. Inte

rval Grade 4

STAR Early Literacy, Grades K-4

OLSSGP

Adding Winter Scores as Covariates with SGP

• Using 80% interval width curves as a baseline, we can further examine how much the intervals are reduced when adding a winter assessment

• The decline in the 80% interval can be used as an indicator of the added precision provided by the winter assessment

STAR Math Example

STAR Reading Example

STAR Early Literacy Example

Some Examples of STAR Score Patterns

Subject Fall Winter Spring Change in 80% Interval

MATH 501 32 531 (+432)625 91 790 (+321)

3 833 852 (-208)2 846 771 (-209)

READING 392 891 589 (+41)376 915 551 (+40)984 1346 1333 (-148)928 1345 1255 (-256)

LITERACY 835 315 832 (+297)777 312 840 (+282)328 858 823 (-242)306 829 851 (-256)

0 200 400 600 800 1000

02

04

06

08

01

00

Fall Score

De

clin

e in

Siz

e o

f 80

% C

on

f. In

terv

al

Grade 5

STAR Math Example

OLS

SGP

0 400 800

20

40

60

80

120

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 1

0 200 60020

40

60

80

100

120

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 2

0 400 800

20

60

100

140

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 3

0 400 800 1200

20

40

60

80

100

120

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 4

0 400 800

20

40

60

80

100

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 5

0 400 800

20

40

60

80

120

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 6

STAR Math, Grade 1-6

OLS SGP

0 400 800 1200

20

40

60

80

120

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 7

0 400 80020

40

60

80

100

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 8

200 600 1000

20

40

60

80

100

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 9

0 400 800

50

100

150

200

250

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 10

200 600 1200

50

100

150

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 11

200 600 1000

050

100

150

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 12

STAR Math, Grades 7-12

OLS SGP

STAR Reading Example

OLS

SGP

0 400 800

30

40

50

60

70

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 1

0 400 80030

40

50

60

70

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 2

0 400 800

20

40

60

80

100

120

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 3

0 400 800

25

30

35

40

45

50

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 4

0 400 800 1400

30

35

40

45

50

55

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 5

0 400 800 1400

35

40

45

50

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 6

STAR Reading, Grades 1-6

OLS SGP

0 400 800 1400

36

38

40

42

44

46

48

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 7

200 600 120020

25

30

35

40

45

50

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 8

0 400 800 1400

10

20

30

40

50

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 9

0 400 800 1400

-20

020

40

60

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 10

200 600 1200

10

20

30

40

50

60

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 11

200 600 1200

010

20

30

40

50

60

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 12

STAR Reading, Grades 7-12

OLS SGP

300 400 500 600 700 800 900

02

04

06

08

01

00

12

0

Fall Score

De

clin

e in

Siz

e o

f 8

0%

Co

nf. In

terv

alGrade 1

STAR Early Literacy Example

OLS

SGP

300 500 700 900

20

30

40

50

60

70

80

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade K

300 500 700 900

020

40

60

80

120

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 1

300 500 700 900

020

40

60

80

120

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 2

400 600 800

020

40

60

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 3

400 600 800

-10

010

20

30

40

50

Fall Score

Declin

e in

Siz

e o

f 80%

Conf. Inte

rval

Grade 4

STAR Early Literacy, Grades K-4

OLS SGP

Evaluating Predicted Spring Scores in Terms of State Proficiency Thresholds

• The accuracy of SGP and OLS predictions can also be compared against the thresholds associated with state-specific proficiency categories

• By assuming normally distributed residuals (with constant variance) for OLS, SGP and OLS can each be used to define a probability that the spring score will exceed a predefined threshold

Academic Year

ST

AR

Ma

th S

ca

led

Sco

re

**

95 %ile

80 %ile

50 %ile

20 %ile

5 %ile

Fall Winter Spring

50

06

00

70

08

00

90

01

00

0

ProficiencyThreshold

Example of SGP Percentile Cuts against State-Defined Proficiency Category

MATH READING

GradeProficiency

Cuta Percentile OLS R2,b SGP R2,bProficiency

Cuta Percentile OLS R2,b SGP R2,b

3 617 48 .40 .41 445 48 .53 .66

4 709 61 .40 .40 531 52 .53 .68

5 758 60 .42 .42 592 49 .52 .69

6 810 65 .46 .46 647 42 .52 .70

7 823 74 .48 .49 825 51 .55 .74

8 859 68 .47 .48 933 53 .55 .73

aEstimated STAR cutscores for the Tennessee Comprehensive Assessment Program (TCAP)bEfron’s Pseudo R2

Comparing SGP and OLS on Accuracy of Proficiency Predictions

Extending SGP to Accommodate Multiple Intermediate Assessments

• How can additional intermediate assessments be used in SGP to further improve predicted spring scores?•Challenge: Handling varying-time point assessment schedules•One possible solution: Linear interpolation to fixed node locations

Conclusions and Future Directions

• Our SGP analyses suggest substantial variability in the precision of spring score predictions for STAR Math, Reading and Early Literacy depending on fall scores

• There is clear value in incorporating winter assessments into SGP---the largest value occurs for students with extreme fall scores in STAR Math, intermediate fall scores in STAR Reading

• More experimentation needed to determine how best to make use of multiple intermediate assessments within SGP