Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard...
-
Upload
jude-brown -
Category
Documents
-
view
219 -
download
2
Transcript of Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard...
![Page 1: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/1.jpg)
Setting Cutoff Scores in C4
Click on the screen or press the right arrow key () on your keyboard to view the next slide. Press the left arrow key () to go back to the previous slide.
There is no audio accompanying this presentation
Copyright © 2009 Biddle Consulting Group, Inc.
![Page 2: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/2.jpg)
This presentation is provided as part of our commitment to serving our clients
Please send comments or suggestions to [email protected]
CritiCall® is a registered trademark of Biddle Consulting Group, Inc.
800-999-0438
![Page 3: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/3.jpg)
General Notes• Cutoff scores in C4 are set based on
AutoTest Codes– AutoTest Codes are a series of test modules that
an employer groups together for testing for a specific job or job category
– You must create an AutoTest Code using the Test Creation Wizard (or the AutoTest Setup feature) before cutoff scores for that AutoTest code can be set
• Cutoff scores are typically determined by an employer’s own “subject matter experts” using C4’s Validation Wizard feature– They are then entered using the “Set Cutoff
Scores” feature in the C4 Administrator’s Program
![Page 4: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/4.jpg)
Table of Contents Quick Links
Click here with your mouse to continue the presentation about setting cutoff scores
Click here to jump ahead in the
presentation to the section on using Cutoff Scores and
Printing Candidate Score
Reports with Pass or Fail Status
Indicated
![Page 5: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/5.jpg)
Before you set cutoff scores …
• We strongly recommend that you review the validation report you created before you set any cutoff scores to make certain all of the information has been entered appropriately by your job experts
![Page 6: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/6.jpg)
Before you set your cutoff scores you should check to see whether you have used the
Validation Wizard feature for the AutoTest Code in question
To do this, click on the “Validation Report” link in the C4 Administrator’s Program
![Page 7: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/7.jpg)
Select the AutoTest Code you wish to view from the Validation Report menu
![Page 8: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/8.jpg)
If you have not validated this AutoTest Code using the Validation Wizard you will see this message
![Page 9: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/9.jpg)
If you have validated this AutoTest Code using the Validation Wizard you will see this screen
You are now ready to set your cutoff scores.
Hint: It may be helpful to print the Validation Report before you set the cutoff scores
777777
After 7 to 10 Job Experts have used the Validation Wizard, it should say “Yes” for all of the tests. If not, you will need to create a new AutoTest Code
without the invalid tests
It is recommended that a minimum of 7 to 10 Job Experts
use the Validation Wizard to validate the tests, if possible
![Page 10: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/10.jpg)
Click here
To set the cutoff scores, you first click on the “Set Cutoff Scores” link on the C4 Administrator’s Screen
![Page 11: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/11.jpg)
Default AutoTest Code initially appears here –
Use this menu to view cutoff score settings
for other AutoTest Codes
This is the screen you see if the listed AutoTest Code has been validated using the “Validation Wizard” feature
![Page 12: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/12.jpg)
“Call Center”
Note: This message will also appear if Subject Matter Experts have responded to the Validation Wizard survey for the listed AutoTest
Code, but the Validation Report feature has not been run in the Administrator’s Program for that AutoTest Code
The message below appears on the screen if no Validation Report has been run for the chosen AutoTest Code
![Page 13: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/13.jpg)
The first column shows the tests contained in this AutoTest Code
![Page 14: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/14.jpg)
The second column is where you set the cutoff scores for each of the tests
![Page 15: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/15.jpg)
If a Validation Report has been run for this AutoTest Code you have five choices when setting cutoff scores…
First choice: Keep the cutoff score at “none,” which is the default.
This means the cutoff score is set at zero – everyone passes no matter what they score.
Default is “None”
![Page 16: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/16.jpg)
Second choice: Set the cutoff score at the cutoff score suggested by your subject matter experts less one standard error of measurement
Less 1 SEM
![Page 17: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/17.jpg)
The cutoff scores listed in the “Cutoff Score Setup” area match the cutoff
scores shown in the Report Summary section of the C4 Validation Report
Summary
This is a sample of the Report Summary section of a C4Validation Report
Where do the cutoff scores in the “Cutoff Score Setup” screen come
from?
![Page 18: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/18.jpg)
Question: Why is there more than one recommend cutoff score for each test?
Answer: Because no tests are perfect. They include "measurement error" and the U.S. Supreme Court has supported
adjusting minimum passing (cutoff) scores to account for this.
U.S. v. South Carolina, 434 US 1026 (1978)
![Page 19: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/19.jpg)
The U.S. Supreme Court has supported that when a test is used for selection (i.e., hiring) purposes, employers consider adjusting the cutoff score downward one, two, or three standard error of
measurements (SEMs) considering the five factors below
![Page 20: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/20.jpg)
The standard error of measurement for each test is
shown in the Detailed Validation Report
What is a Standard Error of Measurement?
If a person were to take the same test repeatedly, with no change in his or her level of knowledge and preparation, it is possible that some of the resulting scores would be slightly
higher or slightly lower than the score that precisely reflects the person’s actual level of knowledge and ability. The Standard Error of Measurement sets a boundary around where
these hypothetical scores will lie. For example, a person who scores 45 on a typing test that has a SEM of 5 would most likely score between 40 and
50 about 70% of the time.
![Page 21: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/21.jpg)
If you had to compute the standard error of measurement yourself, you would use
this formula
• SEM refers to the standard error of measurement
• SD refers to the standard deviation of test scores
• rxy refers to the reliability of the testNote: More sophisticated methods are available for computing conditional SEMs (SEMs that change throughout the test score range) for tests that have more test items than are typically contained on
CritiCall tests.
![Page 22: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/22.jpg)
• SEM refers to the standard error of measurement
• SD refers to the standard deviation of test scores
• rxy refers to the reliability of the test
For more details, refer to the “Test Validation” chapter, “Cutoff Scores” section, of the CritiCall Administrator’s Manual
The C4 System automatically computes the
standard error of measurement and adjusted cutoff scores so
you don’t have to
If you had to compute the standard error of measurement, you would use this
formula
![Page 23: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/23.jpg)
For example, the average less one SEM for this test is
the average of suggested cutoffs (65%) minus one
SEM (1.718), which is rounded out to 63%.
The average less two SEMs for this test is
the average of suggested cutoffs (65%) minus two SEMs (3.435), which is
rounded out to 62.
We follow the same logic for the average less three SEMs
Calculating the adjusted cutoff
scores using the standard error of
measurement
![Page 24: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/24.jpg)
So, the U.S. Supreme Court recommends you consider
using one of these three cutoff scores instead of the
average of suggested cutoffs
Please contact Biddle Consulting Group, Inc. at 800-999-0438 if you have additional questions about how cutoff scores are indicated in the
Validation Report
![Page 25: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/25.jpg)
Third choice: Set the cutoff score at the cutoff score suggested by your subject matter experts less two standard error of measurements
Less 2 SEMs
![Page 26: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/26.jpg)
Fourth choice: Set the cutoff score at the cutoff score suggested by your subject matter experts less three standard error of measurements
Less 3 SEMs
![Page 27: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/27.jpg)
Add cutoff score
Fifth choice: Set a custom cutoff score.
Be extremely careful when setting custom cutoff scores since you may increase your liability exposure if you set an inappropriate cutoff score
![Page 28: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/28.jpg)
Default is “None”
If no Validation Report has been run for this AutoTest Codeyou will have only two choices
First choice: Keep the cutoff score at “none,” which is the default.
This means the cutoff score is set at zero – everyone passes no matter what they score.
![Page 29: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/29.jpg)
If no Validation Report has been run for this AutoTest Code
We recommend entering one of the adjusted cutoff scores (average cutoff score minus one, two, or three standard errors of measurement)
listed in the Validation Report you have produced for this AutoTest Code
Second choice: Set the cutoff score at the cutoff score suggested by your subject matter experts or using some custom cutoff score
Add cutoff score
![Page 30: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/30.jpg)
Rule: Pass/Fail
If a test taker scores below the cutoff score that has been set for ANY test that uses the “Pass/Fail” rule, that test taker fails that test. This rule
currently cannot be changed or modified.
The third column shows how the cutoff score will be used
![Page 31: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/31.jpg)
“Low SME Warning” is shown if less than 7 SMEs used the
Validation Wizard for this test
The final column indicates the total number of the subject matter experts (SMEs) who have used the Validation
Wizard for this AutoTest Code
Number of Subject Matter Experts
The U.S. Courts indicate that a minimum of seven to ten Subject Matter Experts should provide validation information, if possible
![Page 32: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/32.jpg)
After all of the cutoff scores are set, click on the “Save” and then the “Close” buttons
If you wish to exit without saving any changes, just click on the “Close” button
![Page 33: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/33.jpg)
Using Cutoff Scores and Printing Candidate Score
Reports with Pass or Fail Status Indicated
![Page 34: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/34.jpg)
Click here
To print a Candidate Score Report showing test takers Pass or Fail status, click on the “Candidate Score Report” link on the C4
Administrator’s Screen
![Page 35: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/35.jpg)
First, select the candidate(s) whose Candidate Score Report you wish to print by clicking on the check box by their names
Hint: You can limit the candidates who appear on this screen by creating “filters” in the area provided (see above).
After filtering, click on the “Update List” button to apply the filter. (See CritiCall Manual for more details)
![Page 36: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/36.jpg)
To create a report that indicates whether a person passes or fails an AutoTest Code you have set cutoff scores for, check () the “Pass/Fail
Report” option on the Candidate Score Report dialogue box before printing the report
![Page 37: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/37.jpg)
The dialogue box shown here will appear on the screen even if you have used the Validation Wizard for this AutoTest Code. It is merely an
advisory message reminding users that they must set the cutoff scores before they will accurately appear on the Pass/Fail version of
the Candidate Score Report.
![Page 38: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/38.jpg)
This is what the Pass/Fail Candidate Score Report looks like
IMPORTANT! Check the AutoTest Code name here to make sure this is the AutoTest Code you wanted
to score
If this is not the AutoTest Code you wanted to score, it may be beneficial to go back to the Candidate Score Report dialogue box and
limit the test takers by filtering
![Page 39: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/39.jpg)
This is what the Pass/Fail Candidate Score Report looks like
Test Taker’s score for each test
The tests in the Candidate Score Report are listed in the order in which they were
administered to the test taker
![Page 40: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/40.jpg)
This is what the Pass/Fail Candidate Score Report looks like
Cutoff scores you have set using the Set Cutoff
Score feature
Candidates’ Pass/Fail status for each test
![Page 41: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/41.jpg)
This is what the Pass/Fail Candidate Score Report looks like
Pass in this column means the test taker scored at or above the cutoff score set for each of the C4 test administered
Fail in this column means the test taker scored below the cutoff score set for a C4 test
Pass/Fail status for each test module and for the
overall scores
![Page 42: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/42.jpg)
This is what the Pass/Fail Candidate Score Report looks like
In this example the test taker failed the “Customer Service Interaction” test since they scored (61%), which is lower than the
cutoff score (63%) that had been set
![Page 43: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/43.jpg)
In our experience…
• Many employers find it beneficial to consider only those candidates who have passed all of the required tests in the AutoTest Code– Those who fail one or more tests may
have difficulties in those areas if hired– Those who fail one or more tests are
more likely to need additional supervision and training if hired
![Page 44: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/44.jpg)
Frequently Asked Question
• Question: Can I change the cutoff scores for an AutoTest Code after they have been set and used to score some test takers?
• Answer: Yes. You can change cutoff scores at any time. – However, every Candidate Score Report you
print out reflects the most recently set cutoff scores, even if some of those candidates have been graded using previously set cutoff scores
![Page 45: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/45.jpg)
If you need assistance or more information
• Technical Support – 800-999-0438 ext 127• Sales – 800-999-0438 ext 150• Other Assistance – 800-999-0438 ext 151
• Or email [email protected]
C4 (CritiCall® for Contact Centers) is a product of Biddle Consulting Group, Inc. 193 Blue Ravine Road, Suite 270, Folsom, CA 95630
CritiCall® is a registered trademark of Biddle Consulting Group, Inc.
![Page 46: Setting Cutoff Scores in C 4 Click on the screen or press the right arrow key ( ) on your keyboard to view the next slide. Press the left arrow key ( )](https://reader035.fdocuments.us/reader035/viewer/2022081518/5518c538550346881f8b57c5/html5/thumbnails/46.jpg)
We hope this presentation has been helpful to you
Please call us at 800-999-0438 or email us at [email protected] if you have comments or suggestions about this presentation; or if there are other presentations you would
like to see in the future
Copyright © 2009 Biddle Consulting Group, Inc.