Establishing MME and MEAP Cut Scores Consistent with College and Career Readiness

A study conducted by the Michigan Department of Education (MDE) and ACT, Inc.

Overview of MEAP and MME

MEAP (Michigan Educational Assessment Program) MME (Michigan Merit Examination)

3 4 5 6 7 8 9 11Mathematics MEAP MEAP MEAP MEAP MEAP MEAP MMEReading MEAP MEAP MEAP MEAP MEAP MEAP MMEWriting MEAP MEAP MMEScience MEAP MEAP MMESocial Studies MEAP MEAP MME

GradeContent Area

Why Reset Cut Scores Now?

Disjoint cut scores Relative difficulty of cut scores increases by grade in grades 3-8 Relative difficulty spikes in high school (grade 11)

Relatively low cut scores Based on standard setting panelists’ interpretation of “basic skills”

New economy Requires a redefinition of “basic skills” Need students to be prepared for technical career training or college

State Board of Education History Adopted new, rigorous high school content standards in 2006 Adopted new high school exam in 2006 Did not want to adopt college ready cut scores at that time (wanted to give time

for schools to implement the new content standards) Requested that in five years time the issue be revisited to raise cut scores

Previous Impact Data

Purpose of the study Set new cut scores representing being on track to career and college readiness in the

grades and subjects in red

Not needed in writing, since the writing standards were originally set to be consistent with ACT’s college readiness benchmarks

Need three cut scores to create four performance levels Advanced Proficient Partially Proficient Not Proficient (no cut score needed)

3 4 5 6 7 8 9 11Mathematics MEAP MEAP MEAP MEAP MEAP MEAP MMEReading MEAP MEAP MEAP MEAP MEAP MEAP MMEWriting MEAP MEAP MMEScience MEAP MEAP MMESocial Studies MEAP MEAP MME

GradeContent Area

Standard Setting

Reasoned Judgment Subjective (panelist judgment) With objective elements (e.g., ordered item booklets)

Empirical Objective

External criterion (grade in related freshman course) was not created for the purpose of standard setting

With subjective elements External criterion (freshman college course grade) is itself a subjective judgment on the part

of the college professor

Choice Empirical Meaning

Two types of linkages were needed to

identify the cut scores:

Link 11th grade MME scores to freshman college grades to identify cut scores on the MME.

Link MEAP scores to MME scores to identify cut scores on one or more grades of the MEAP.

Step 1: The MME

To do listDefine what is meant by college successCollect data on students with MME scores

and college outcomesDecide on a method that links MME scores

and college success

Measure of college success

First year courses Course grade- either average grade, or

specific grade or betterExamples:average grade of 3.0Proportion of students with a grade of B or

better

The MME - Data

MME Test data available with MI student ID

CEPI data available with MI student ID

This is a very good thing

The MME data

College grade data taken from Fall 2009 and Fall 2010

Course grade data from 39 public colleges 19 4-year institutions 20 2-year institutions

Each college was given the opportunity to define their “first year courses” in each course area

The MME - data

Number of courses varied by subject

Mathematics1 course

College Algebra

Science4 courses

Biology/Life ScienceGeneral Chemistry

PhysicsOther

The MME – College Grade Data

Total sample sizes:Reading 40,164Mathematics 6,567Science 16,180Social Science 41,780

>100,000 records

The MME - Methods

Signal Detection Theory (SDT) Logistic Regression (LR) Conditional mean

Very Similar

The SDT Method

College grade

Cut score(To be

determined)

Successful/ Successful

Not Successful/ Not Successful

Not Successful/ Successful

Successful/ Not Successful

The Logistic Method

0

0.2

0.4

0.6

0.8

1

1050 1070 1090 1110 1130 1150

MME Score

Pro

b. o

f su

cces

s

0.5

Cutscore

The MME

To get a cutscore, we must choose a success criterion and a methodB or higher gradeSDT method

MME – the results

Subject Proficient Cut Score

Past Proficient Cut Score

Mathematics 1116 1100

Reading 1108 1100

Science 1126 1100

Social Science 1129 1100

Step 2: MEAP to MME

What does it mean to say that a 5th grade student is proficient?

General answer: A student is CCR if his/her score in grade 5 indicates that the student is on track to be college ready in grade 11

College Readiness in Grade 5

Score on Grade 11 test

Sco

re o

n G

rade

5 te

st

College ReadyNot College Ready

What link to make?

Option 1 – Link Grade 8 to Grade 11,

Grade 7 to Grade 8, etc.

Option 2 – Link Grade 8 to Grade 11,

Grade 7 to Grade 11, etc

Advantage: Less Measurement Error

Advantage: Larger Sample Size

Cohort with at Least Two Years of Data

Cohort

Grade Sample Size

3 4 5 6 7 8 9 10a 11 12

N_Totalb

N_Full_Matchedc

1 09-10 10-11 - - - - - - - - 118,468 107,844

2 08-09 09-10 10-11 - - - - - - - 123,896 105,019

3 07-08 08-09 09-10 10-11 - - - - - - 128,557 101,277

4 06-07 07-08 08-09 09-10 10-11 - - - - - 135,268 98,131

5 05-06 06-07 07-08 08-09 09-10 10-11 - - - - 140,729 94,044

6 - 05-06 06-07 07-08 08-09 09-10 10-11 - - - 149,950 93,447

7 - - 05-06 06-07 07-08 08-09 09-10 10-11 - - 151,047 98,155

8 - - - 05-06 06-07 07-08 08-09 09-10 10-11 - 161,869 88,526

9 - - - - 05-06 06-07 07-08 08-09 09-10 10-11 163,246 93,004

10 - - - - - 05-06 06-07 07-08 08-09 09-10 160,813 97,172

11 - - - - - - 05-06 06-07 07-08 08-09 158,255 104,352

Fewer links – smaller(?) sample

MME scoresGrade 11

MEAP scoresGrade 7Grade 8Grade 9

MEAP scoresGrade 3Grade 4Grade 5Grade 6

MEAP – the results (mathematics)

Grade New Cut Score Old Cut Score

8 830 800

7 731 700

6 629 600

5 531 500

4 434 400

3 338 300

Validity checks

Consistency of classification across yearsShould be high, but not too high

Consistency of classification with different tests (like the ACT)Note that MME contains items from the ACT

as one of its constituent parts for Reading, Math, and Science

Classification Consistency Rates Year to Year

Mathematics

11-grade - 65% -

8-11 83% 86% 95%

7-8 81% 84% 95%

6-7 82% 83% 96%

5-6 81% 84% 95%

4-5 80% 82% 94%

3-4 77% 80% 95%

GradePartiallyProficient Proficient Advanced

Consistency of MME with ACT

   

ACT Math Meeting College and Career

Readiness Benchmarks

   Yes No

MME Math

Proficient

Yes30,53425.5%

1,2411.0%

No7,0045.8%

81,04167.6%

93%

Consistency of MME with ACT

Subject Consistency Rate

More difficult

Mathematics 93% MME

Reading 86% ACT

Science 90% ACT

Consequences – Reading(Student Level)

Consequences – Reading(Elementary and Middle School Level)

0

20

40

60

80

100

120

140

160

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Num

ber o

f Sch

ools

Percent Proficient or Advanced

With New Cut Scores

With Old Cut Scores

Consequences – Reading(High School Level)

0

5

10

15

20

25

30

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Num

ber o

f Sch

ools

Percent Proficient or Advanced

With New Cut Scores

With Old Cut Scores

Consequences – Mathematics (Student Level)

Consequences – Mathematics(Elementary and Middle School Level)

0

20

40

60

80

100

120

140

160

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Num

ber o

f Sch

ools

Percent Proficient or Advanced

With New Cut Scores

With Old Cut Scores

Consequences – Mathematics(High School Level)

0

10

20

30

40

50

60

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100

Num

ber o

f Sch

ools

Percent Proficient or Advanced

With New Cut Scores

With Old Cut Scores

Message

Michigan schools are not adequately preparing students for freshman-level college/career-training courses

Michigan schools are better preparing students for entry-level reading-heavy (humanities, social science) courses than for entry-level mathematics courses

Message

Florida recently attempted to do this in writing, but the increase in cut scores was rejected

We believe Michigan was able to successfully make the transition to more rigorous cut scores because of advance notice to the field, the press, and the state board of education regarding the anticipated impact and a strong argument for the need to revisit cut scores

Thank you for your attention

Any Questions or Comments?