Setting Accuplacer Cut Scores Setting Accuplacer Cut Scores for WritePlacer Plus and for WritePlacer Plus and

College MathCollege Math

Becky J. Mussat-Whitlow, M.A., Ph.D.Becky J. Mussat-Whitlow, M.A., Ph.D. Director of Institutional Assessment Director of Institutional Assessment

Winston-Salem State UniversityWinston-Salem State University

Robert Ussery, M.S.Robert Ussery, M.S.Assistant Vice Chancellor for Academic Affairs Assistant Vice Chancellor for Academic Affairs

North Carolina A&T State University North Carolina A&T State University

18th Annual Accuplacer National Conference18th Annual Accuplacer National Conference Fort Lauderdale, FloridaFort Lauderdale, Florida

June 26, 2008June 26, 2008

IntroductionIntroduction

• Historical contextHistorical context Local bubble sheet test for math placementLocal bubble sheet test for math placement ETS’s online Criterion for English placementETS’s online Criterion for English placement

• Executive decision to use AccuplacerExecutive decision to use Accuplacer For math and English placementFor math and English placement Single vendorSingle vendor

• November, 2007November, 2007 Assign project managerAssign project manager Complete project planComplete project plan

• December, 2007December, 2007 – Secure funding – Secure funding

• January, 2008January, 2008 Provost appoint Placement Committee Provost appoint Placement Committee Recruit cut score study consultantRecruit cut score study consultant

• February, 2008February, 2008 Appoint cut score study panels and chairsAppoint cut score study panels and chairs Conduct Round 1 of student testingConduct Round 1 of student testing

• March, 2008March, 2008 Conduct Cut Score study during Spring Break (March 3 – 7)Conduct Cut Score study during Spring Break (March 3 – 7) Round 2 of student testingRound 2 of student testing Cut score recommendations to Placement CommitteeCut score recommendations to Placement Committee

Project TimelineProject Timeline

The standard setting process should pay careful The standard setting process should pay careful attention toattention to::

1.1. Selection of panelists Selection of panelists

2.2. Training Training

3.3. Aggregation of data into a final set of standards Aggregation of data into a final set of standards

4.4. Validation of performance standards Validation of performance standards

5.5. Careful documentation of the process. Careful documentation of the process.

Reference: Hansche, L.N. (1998). Reference: Hansche, L.N. (1998). Handbook for the development of performance Handbook for the development of performance standardsstandards. Bethesda, MD: US Department of Education, Council of Chief . Bethesda, MD: US Department of Education, Council of Chief State School Officers.State School Officers.

Creating Defensible Cut ScoresCreating Defensible Cut Scores

At what point along the scale should the At what point along the scale should the passing mark be set?passing mark be set?

Critical QuestionCritical Question

Cut scores split a continuous distribution of Cut scores split a continuous distribution of knowledge, skills, and abilities into separate knowledge, skills, and abilities into separate regions.regions.

Need to determine the preference for classification Need to determine the preference for classification error.error. Do you prefer to pass students who deserved to fail?Do you prefer to pass students who deserved to fail?

OROR

Do you prefer to fail students who deserved to pass?Do you prefer to fail students who deserved to pass?

Classification ErrorClassification Error

• 2000 BC – Chinese Military Selection2000 BC – Chinese Military Selection

• 1800s - Integration of Psychology and Statistics1800s - Integration of Psychology and Statistics

• 1970s – Mandated pupil proficiency testing1970s – Mandated pupil proficiency testing

• 1978 Journal of Educational Measurement1978 Journal of Educational Measurement

• 1982 ETS 1982 ETS Passing ScoresPassing Scores publication publication

History of Standard SettingHistory of Standard Setting

““People have been setting cut scores People have been setting cut scores for thousands of years, but it is only for thousands of years, but it is only since the middle of the 20since the middle of the 20thth century that century that measurement professionals began to measurement professionals began to pay much attention.”pay much attention.”

~ From: Cizek, G.J. (2001). ~ From: Cizek, G.J. (2001). Setting Setting Performance StandardsPerformance Standards. Hillsdale, NJ: . Hillsdale, NJ: Lawrence Erlbaum Associates. (page 20)Lawrence Erlbaum Associates. (page 20)

Overview of Standard Setting

All standard setting methods involve All standard setting methods involve judgmentjudgment

Performance standards (cut scores) Performance standards (cut scores) may be set too high or low may be set too high or low

Need a reasonable process to arrive at Need a reasonable process to arrive at decisiondecision

Two Categories of Setting Performance Standards

Test-based methods – methods in Test-based methods – methods in which panelists are focused on a which panelists are focused on a review of test contentreview of test content

Student based methods – methods Student based methods – methods that focus on studentsthat focus on students

Guiding Principles to Ensure Fairness in Establishing Cut Scores

Those who set the standards should be thoroughly Those who set the standards should be thoroughly knowledgeable about the content domain that is to be knowledgeable about the content domain that is to be assessed, the population of examinees who are to assessed, the population of examinees who are to take the assessment, and the uses to which the take the assessment, and the uses to which the results are to be put. (p. 316)results are to be put. (p. 316)

Proficiency classifications should have the same Proficiency classifications should have the same meaning for all sub-groups. (p. 317)meaning for all sub-groups. (p. 317)

~ From: Bond, L. (1995). Ensuring fairness in the setting of ~ From: Bond, L. (1995). Ensuring fairness in the setting of performance standardsperformance standards. . InIn Proceedings of Joint Conference Proceedings of Joint Conference on Standard Setting for Large-Scale Assessments (pp. 311-on Standard Setting for Large-Scale Assessments (pp. 311-324). 324). Washington, DC: National Assessment Governing Washington, DC: National Assessment Governing Board and National Center for Education Statistics. Board and National Center for Education Statistics.

Guiding Principles to Ensure Fairness in Establishing Cut Scores

If the assessment is to be used as a screen for future If the assessment is to be used as a screen for future educational opportunities, the content of the assessment educational opportunities, the content of the assessment and the level of proficiency required should be and the level of proficiency required should be demonstrably related to future success. (p. 317)demonstrably related to future success. (p. 317)

Attention must be paid to the consequences of particular Attention must be paid to the consequences of particular uses of an assessment. (p. 318)uses of an assessment. (p. 318)

~ From: Bond, L. (1995). Ensuring fairness in the setting of ~ From: Bond, L. (1995). Ensuring fairness in the setting of performance standardsperformance standards. . InIn Proceedings of Joint Conference on Proceedings of Joint Conference on Standard Setting for Large-Scale Assessments (pp. 311-324). Standard Setting for Large-Scale Assessments (pp. 311-324). Washington, DC: National Assessment Governing Board and Washington, DC: National Assessment Governing Board and National Center for Education Statistics.National Center for Education Statistics.

Guiding Principles to Ensure Equity in the Process of Setting Cut Scores

Adequate notice of proposed actions.Adequate notice of proposed actions. Ample provision of opportunities for participationAmple provision of opportunities for participation Adequate records of all discussions and decisions by the Adequate records of all discussions and decisions by the

participants.participants. Timely distribution of minutes and ballot resultsTimely distribution of minutes and ballot results Careful attention to minority opinions.Careful attention to minority opinions.

~ From: Collins,B. L. (1995). The consensus process in standards ~ From: Collins,B. L. (1995). The consensus process in standards developmentdevelopment. . InIn Proceedings of Joint Conference on Standard Proceedings of Joint Conference on Standard Setting for Large-Scale Assessments (pp. 203-220). Setting for Large-Scale Assessments (pp. 203-220). Washington, Washington, DC: National Assessment Governing Board and National Center DC: National Assessment Governing Board and National Center for Education Statistics.for Education Statistics.

Common Problems Use of ambiguous descriptions of Use of ambiguous descriptions of

performance standardsperformance standards Failure to involve key stakeholders in Failure to involve key stakeholders in

standard setting processstandard setting process Failure to devote sufficient time to Failure to devote sufficient time to

establish cut scoresestablish cut scores Failure to document the processFailure to document the process Failure to validate the processFailure to validate the process

Questions to Consider

How many cut scores are needed?

Two (1 to differentiate between remedial and regular, 1 to differentiate between regular and advanced)

VERSUS

One (to differentiate between remedial and regular course placement)

Desirable Characteristics of Cut Scores

Should be understandable and useful for all Should be understandable and useful for all stakeholdersstakeholders

Clearly differentiate among levelsClearly differentiate among levels

Grounded in student workGrounded in student work

Built by consensusBuilt by consensus

Focus on learningFocus on learning

Contrasting Group Method

A group of panelists qualified to assess the A group of panelists qualified to assess the content domain and students being assessed content domain and students being assessed are asked to classify students into two groups are asked to classify students into two groups (master vs. non-masters)(master vs. non-masters)

Panelists initial make judgments regarding Panelists initial make judgments regarding grouping of students. Then, the performance grouping of students. Then, the performance of examinees empirically determined.of examinees empirically determined.

Contrasting Group Method

Two distributions created to represent the Two distributions created to represent the students’ actual (obtained) scores on the students’ actual (obtained) scores on the assessment separately; one for those assessment separately; one for those students judged to have acceptable skills by students judged to have acceptable skills by the standard setters and another for those the standard setters and another for those students whose performances were judged to students whose performances were judged to be unacceptable.be unacceptable.

The point at which the two distributions The point at which the two distributions intersect may be chosen as the cut score intersect may be chosen as the cut score location.location.

Hypothetical Illustration of Contrasting Groups Distributions

NonmastersDistribution

MastersDistribution

Score Scale

Cχ

ƒ

Modified Contrasting Group Method

Students were classified into three broad Students were classified into three broad performance categories of average, below performance categories of average, below average, or above average and administered average, or above average and administered the placement tests.the placement tests.

Score distributions were plotted to represent Score distributions were plotted to represent the students’ actual (obtained) scores on the the students’ actual (obtained) scores on the assessment separatelyassessment separately

The distribution plot was visually analyzed to The distribution plot was visually analyzed to identify an appropriate cut score.identify an appropriate cut score.

Modified Contrasting Group Method

Panelists completed the placement test role Panelists completed the placement test role playing as a student with average and above playing as a student with average and above average abilityaverage ability

Score distributions were plotted to represent Score distributions were plotted to represent the panelists’ actual (obtained) scores the panelists’ actual (obtained) scores

The distribution plot was visually analyzed to The distribution plot was visually analyzed to help guide the establishment of an help guide the establishment of an appropriate cut score.appropriate cut score.

Multi-Step Approach to Establish Cut Scores

Panelists created performance level descriptions Panelists created performance level descriptions (PLDs) and mapped these PLDs to College Board (PLDs) and mapped these PLDs to College Board proficiency level statements   proficiency level statements  

Panelists reviewed student score distributionsPanelists reviewed student score distributions Panelists completed placement test and reviewed Panelists completed placement test and reviewed

their score distributions.their score distributions.

Additionally, for English, panelists retrospectively categorized Additionally, for English, panelists retrospectively categorized papers into 2 groups (remediation required vs. no remediation papers into 2 groups (remediation required vs. no remediation required) and used this information to guide cut score required) and used this information to guide cut score establishment.establishment.

          

Three Day Panel Sessions

Panels met for 3 hour sessions on 3 Panels met for 3 hour sessions on 3 consecutive days.consecutive days.

Math Panel: 12 facultyMath Panel: 12 faculty

English Panel: 12 facultyEnglish Panel: 12 faculty

Panel Session I

Panel Session 1: Panel Session 1: Identified the number of cut scores Identified the number of cut scores

needed and the different courses into needed and the different courses into which students would be placedwhich students would be placed

Developed clear and concise Developed clear and concise performance level descriptions for performance level descriptions for each placement level each placement level

Characteristics of Performance Level Descriptions

Written in positive termsWritten in positive terms

Parallel in organization, language, Parallel in organization, language, and styleand style

Written in clear and concise Written in clear and concise language without using language without using unmeasurable qualifiers (e.g., often, unmeasurable qualifiers (e.g., often, seldom, etc.)seldom, etc.)

Panel Session II

Panel Session 2: Panel Session 2: PLDs ApprovedPLDs Approved Initial round of cut score setting Initial round of cut score setting

based upon the results of student based upon the results of student testing and faculty role play testing and faculty role play performanceperformance

For English, student work categorized For English, student work categorized by panelists into 2 groups.by panelists into 2 groups.

Illustration of Score Distributions

Illustration of Score Distributions

1st Cut Score = 30 Students scoring

less than 30 – Remedial

Placement

2nd Cut Score = 42 Students scoring

from 30 to 41 – Math 101

3rd Cut Score = 65 Students scoring from 42 to 64 – Math 110 or

111Students scoring 65 or

higher – Math 131

English Panelist Rating Information

MeanMean SDSD

Remedial Indicated by Two Remedial Indicated by Two or Fewer Panelistsor Fewer Panelists

7.857.85 1.371.37

Remedial Indicated by Three Remedial Indicated by Three or More Panelistsor More Panelists

6.906.90 1.071.07

English Panelist Rating Information

MeanMean SDSD

Remedial Indicated by None Remedial Indicated by None or Some Panelistsor Some Panelists

7.617.61 1.281.28

Remedial Indicated by Remedial Indicated by AllAll PanelistsPanelists

5.05.0 1.011.01

Panel Session III

Panel Session 3: Panel Session 3:

Cut Scores were aligned across Cut Scores were aligned across various methods used.various methods used.

PLDs Mapped To Proficiency Statements

MATH 101MATH 101 PLD: Perform addition, subtraction, PLD: Perform addition, subtraction,

multiplication and division of polynomials; multiplication and division of polynomials; identify factors, terms, constants, variables, identify factors, terms, constants, variables, exponents, and coefficients; and recognize exponents, and coefficients; and recognize graphs of polynomial functions. graphs of polynomial functions.

Lower than ACCUPLACER 40Lower than ACCUPLACER 40 Cut Score 30Cut Score 30

Alignment of Recommendations

Performance level descriptions written by Performance level descriptions written by

NC A & T faulty were mapped to College NC A & T faulty were mapped to College Board proficiency level statements   Board proficiency level statements  

Review score distributions for student Review score distributions for student performance groups to determine cut scoreperformance groups to determine cut score

Cut scores indicated on the basis of the Cut scores indicated on the basis of the student data were considered in conjunction student data were considered in conjunction with performance level descriptions. with performance level descriptions.         

Example of Alignment Process

Based on mapping to Accuplacer

Proficiency Statements. Cut score should fall

between 40 & 63.

Math 110 or 111

Follow-Up Testing and Cut-Score Refinement

Additional Students TestedAdditional Students Tested

Cut Scores RevisedCut Scores Revised

Recommendations Made By PanelistsRecommendations Made By Panelists

Project Timeline after Study March, 2008March, 2008

Committee review and recommendationsCommittee review and recommendations Executive review and policyExecutive review and policy

April, 2008April, 2008 Develop student score validation and upload procedureDevelop student score validation and upload procedure

May, 2008May, 2008 Train proctors and faculty advisorsTrain proctors and faculty advisors Final system testFinal system test

JuneJune Go live with Accuplacer placement testingGo live with Accuplacer placement testing 500 students per week500 students per week

July, 2008July, 2008 Process evaluation and recommendationsProcess evaluation and recommendations

Campus Collaboration

Admissions – New studentsAdmissions – New students Student Affairs – New student orientationStudent Affairs – New student orientation Registrar – Validate student dataRegistrar – Validate student data IT – Upload student data to campus systemIT – Upload student data to campus system Academic Affairs - Proctors and advisorsAcademic Affairs - Proctors and advisors Institutional Research – Management and Institutional Research – Management and

supportsupport

Accuplacer Support

HelplineHelpline

Online procedure guidesOnline procedure guides

Early Results

Five (5) orientation sessions in JuneFive (5) orientation sessions in June Overall, the system worksOverall, the system works Some Accuplacer interface weaknessesSome Accuplacer interface weaknesses

WHERE TO FROM HERE

Develop alternative placement Develop alternative placement measures measures

Implement off-site testingImplement off-site testing

Thank you for attending!

Questions?

NCA&T Support Site

http://qed.ncat.edu/ir&p/availabl.htm