1 PRESENTED TO THE FALL 2011 MEETING OF THE MICHIGAN EDUCATIONAL RESEARCH ASSOCIATION MAY 17, 2011...
-
Upload
louisa-chapman -
Category
Documents
-
view
213 -
download
0
Transcript of 1 PRESENTED TO THE FALL 2011 MEETING OF THE MICHIGAN EDUCATIONAL RESEARCH ASSOCIATION MAY 17, 2011...
1
PRESENTED TO THE FALL 2011 MEETING OF THE MICHIGAN EDUCATIONAL RESEARCH ASSOCIATION
MAY 17, 2011
Methods for Setting Cut Scores to Represent College Readiness
2
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Resources
ContributorsMDE Measurement Research &
Psychometrics UnitACT Research & Development UnitNational Center for Educational
AchievementMichigan Technical Advisory Committee
3
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College Readiness
ACT Methodology Good basis for a starting point What score on the ACT gives an 11th grader a
50/50 chance of obtaining a B or better in a… first-semester of freshman year credit-bearing (non-remedial)
…course in a related subject?
4
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessACT Methodology
ACT Methodology for identifying College Ready Benchmarks http://www.act.org/research/researchers/reports/pdf/ACT_RR2005-3.pdf A logistic regression methodology identifying the point on the ACT subject
score scale where a student has a 50% or better probability of achieving a B or better in the first-semester, credit-bearing, adequately related, freshman course
Start with data from Michigan Public Institutions of Higher Education Identify appropriate credit-bearing freshman courses against which to
analyze the relationship between MME scores and course grades Analyze proportion of students receiving a B or better against MME scale
scores
5
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessACT Methodology
Each dot represents a group of students with a specific test score, plotting the test score against the percent of students with that test score who earned a B or better
6
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessACT Methodology
The curved line is the logistic regression line of best fit through the cloud of points
7
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessACT Methodology
Horizontal line represents a 50% probability of earning a B or better
8
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessACT Methodology
Vertical line represents the test score that give a 50% probability of earning a B or better
9
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessACT Methodology
Issues With the ACT Methodology for the Purpose of Setting Cut Scores on the MME
Non-symmetric, regression-based method Focused on probability prediction
ACT’s purpose Appropriate for ACT’s purpose
Identifying equivalent points on different scales requires the use of a symmetric procedure.
Using a non-symmetric procedure can result in a biased cut score because of regression to the mean when identifying a cut score not at the state average
10
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessProposed Methodology
Proposed Methodology A Method Based on Signal Detection Theory (SDT)
A symmetric method Focused on maximizing classification consistency
Maximizes the percentage of students who are both1. Classified as proficient on 11th grade MME2. Achieved a B or better in their first-semester, freshman-year, credit-
bearing college course Classification consistency is the core of identifying both college
readiness in high school and being on track in lower grades Results in an unbiased cut score because regression to the mean is not
an issue
11
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessProposed Methodology
SDT in a nutshell, for our purposesFocused on the ability to…
Make correct decisions when data contain both “signal” and “noise”
Identify thresholds of “signal” in the data that maximize the ability to make correct classification decisions
12
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessProposed Methodology
Other uses of SDT-based methods germane to our purposes Maximally accurate detection of…
Medical anomalies in reading radiological reports Severe cases in Emergency Room triage Medical impacts in clinical drug trials Objects in naval sonar, military radar, and civilian air traffic control Impact of interventions on memory and cognition
Our purpose Maximally accurate detection of…
College readiness on MME Being on-track to college readiness on MEAP
13
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessProposed Methodology
Basic application of SDT-based methods for our purposes
Using the known outcomes of college course grades, set the MME cut score to… Maximize the number of students who are consistently classified Minimize the number of students who are inconsistently classified
Grade 11 MME(unknown cut
score)
College Grades (known outcome)
Less than B B or Better
Proficient Inconsistent Consistent
Not Proficient Consistent Inconsistent
14
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessProposed Methodology
Start with data from Michigan Public Institutions of Higher Education Identify appropriate first-semester, freshman, credit-bearing (non-
remedial) courses against which to analyze the relationship between MME scores and course grades
Identify students who took both the MME as 11th graders and took those courses
Split those students into those who received a B or higher and those who received a B- or lower
Depicted graphically in the next slides
15
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Each dot on the left side of the vertical line represents one student who achieved a B- or lower in a first-semester freshman year, credit-bearing course
Each dot on the right side of the vertical line represents one student who achieved a B or higher in a first-semester freshman year, credit-bearing course
16
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Each dot on the upper side of the horizontal line represents one student who achieved proficiency on the MME
Each dot under the horizontal line represents one student who was not proficient on the MME
17
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Inconsistent Proficient in 11th grade B- or lower in college
Consistent Not proficient in 11th grade B- or lower in college
Consistent Proficient in 11th grade B or higher in college
18
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
19
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
20
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
75.00% Consistent
21
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
79.69% Consistent
22
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining College ReadinessProposed Methodology
Intend to create three cut scores based on this methodology Produces four performance levels All three cut scores can have externally validated meaning:
Example Cut Score Set 1: A or better in college course as Advanced B or better in college course as Proficient C or better in college course as Partially Proficient
Example Cut Score Set 2: B or better in selective enrollment 4-year Universities as Advanced B or better in non-selective 4-year Universities as Proficient B or better in community college as Partially Proficient
Graphical example shows one cut at a time being set For convenience Statistical methodology can do all three cuts simultaneously
23
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining On-Track for the Highest Grade Level of Each MEAP Subject
From the new MME cut scores, identify new cut scores for MEAP as follows… Mathematics – grade 8 Reading – grade 8 Science – grade 8 Social Studies – grade 9
Use test scores of students who took both the MME in Spring 2010 and the grade 9 MEAP in Fall of 2007 for Social Studies
Use test scores of students who took both the MME in Spring 2010 and the grade 8 MEAP in Fall of 2006 for Mathematics, Reading, and Science
24
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining On-Track for the Highest Grade Level of Each MEAP SubjectEach dot on the graph represents the grade 11
MME score and the score on the highest grade level of MEAP for an individual student
25
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining On-Track for the Highest Grade Level of Each MEAP SubjectMME cut scores
previously identified using SDT-based methods
26
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining On-Track for the Highest Grade Level of Each MEAP Subject
Not Proficienton MME
Partially Proficient on MME
Proficienton MME
Advancedon MME
27
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining On-Track for the Highest Grade Level of Each MEAP Subject
MEAP grade 8 cut scores to be identified
28
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining On-Track for the Highest Grade Level of Each MEAP Subject
29
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining On-Track for the Highest Grade Level of Each MEAP Subject
Adjust the Grade 8 MEAP Cut Scores Up or Down to
Obtain Maximum Classification Consistency
30
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining On-Track for the Remaining Grade Levels of Each MEAP Subject
Use test scores of students who took MEAP in both Fall 2009 and Fall 2010 for Reading and Math
Use test scores of students who took MEAP in both Fall 2007 and Fall 2010 for Science and Social Studies
Why not use the same cohort all the way back to the lowest MEAP grade? Data are not available Need to use the most recent data for identifying each grade level’s
cut score Need to reflect the current state of Michigan education
31
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Defining On-Track for the Remaining Grade Levels of Each MEAP Subject
Use same SDT-based method to go systematically down the line one grade at a time Mathematics and Reading
Connect grade 8 to grade 7 Connect grade 7 to grade 6 Connect grade 6 to grade 5 Connect grade 5 to grade 4 Connect grade 4 to grade 3
Science Connect grade 8 to grade 5
Social Studies Connect grade 9 to grade 6
32
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Next Steps
1. Obtain data from Institutions of Higher Education (currently being done, from CEPI)
2. Determine the appropriate credit-bearing freshman courses against which to analyze MME scores (currently being done, with ACT and CEPI)
3. Identify college-ready cut scores for all subjects on the MME
4. Identify on-track cut scores for all subjects on the MEAP5. Recommend to the State Board of Education the final
cut scores for Fall 2011 MEAP and Spring 2012 MME
33
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
References
Boutis, K. , Pecaric, M. , Seeto, B. and Pusic, M. (2010). Using signal detection theory to model changes in serial learning of radiological image interpretation. Advances in health sciences education: theory and practice, 15(5), 647-58.
Despins, L., Scott-Cawiezell, J., and Rouder, J. (2010). Detection of patient risk by nurses: a theoretical framework. Journal of advanced nursing, 66(2), 465-74.
Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley.Klein, S. and Levi, D. (2009) Stochastic model for detection of signals in noise. Journal of the Optical
Society of America: Optics, image science, and vision, 26(11),110-26.Merlo-Pich, E. and Gomeni, R. (2008). Model-based approach and signal detection theory to evaluate
the performance of recruitment centers in clinical trials with antidepressant drugs. Clinical pharmacology and therapeutics, 84(3), 378-84.
Neal, A. and Kwantes, P. (2009) An evidence accumulation model for conflict detection performance in a simulated air traffic control task. Human factors, 51(2), 164-80.
Pigeau, R., Angus, R., O’Neill, P. and Mack, I. (1995). Vigilance latencies to aircraft detection among NORAD surveillance operators. Human Factors, 37(3), 622-634.
34
College Ready Cut Scores for MEAP and MME • May 17, 2011 • MERA
Contact Information
Joseph A. Martineau, Ph.D. Executive Director Bureau of Assessment & Accountability Michigan Department of Education