Caveon Webinar Series: Lessons Learned from EATP and CSDPTF November 2013
Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014
-
Upload
caveon-test-security -
Category
Education
-
view
162 -
download
0
Transcript of Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014
![Page 1: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/1.jpg)
1
Liz Burns, Juniper NetworksJennifer Semko, Baker & McKenzie
Jack Terry, NBEORachel Schoenig, ACT
Dennis Maynes, Caveon Test Security
Lessons Learned from Using Statistics to Invalidate Scores
June 19, 2014
Caveon Webinar Series presents:
![Page 2: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/2.jpg)
2
Agenda for Today
• Question/Answer Format
• Please use the chat window to ask questions
• Conclusions
![Page 3: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/3.jpg)
3
To Kick Off the Discussion…
Let’s chat about why an organization would use statistics to invalidate scores.
![Page 4: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/4.jpg)
5
Q1
If a candidate hires legal counsel and contests a score invalidation, what are the three most important things you should do?
![Page 5: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/5.jpg)
6
Q1 - Answer
1. Don’t second guess yourself. Remain confident.
2. Be prepared to provide legal counsel with a high level summary.
3. Bring your counsel on board in a timely manner.
If a candidate hires legal counsel and contests a score invalidation, what are the three most important things you
should do?
![Page 6: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/6.jpg)
7
Q2
In your opinion, which statistics provide the most credible information concerning potential test fraud?
![Page 7: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/7.jpg)
8
Q2 - Answer
• Old/New or EVT* Items• Trojan Horse Items• Value• Consistent measurement• Identify candidates who benefit• Can be automated• Easy to explain/understand
In your opinion, which statistics provide the most credible information concerning potential test fraud?
*EVT = Embedded Verification Test
![Page 8: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/8.jpg)
9
Q3
When and how do you believe one should recommend score invalidations to your board or governing body?
![Page 9: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/9.jpg)
10
Q3 - Answer
When? When you have a good faith basis for questioning the validity of a score
How? Provide a full and frank summary; give your Board the tools they need
When and how do you believe one should recommend score invalidations to your board or governing body?
![Page 10: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/10.jpg)
11
Q4
Describe at least three things that should be avoided when using statistics to invalidate scores?
![Page 11: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/11.jpg)
12
Q4 - Answer
1. Using complicated statistical explanations as rationale
2. Using complex charts or graphs to illustrate your point
3. Using the word ‘cheat’, ‘cheater’, or other emotionally-charged, accusatory language
Describe at least three things that should be avoided when using statistics to invalidate scores?
![Page 12: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/12.jpg)
13
Q5
Based on your experience, what are pros and cons to using statistics to invalidate scores?
![Page 13: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/13.jpg)
14
Q5 - AnswerBased on your experience, what are pros and cons to
using statistics to invalidate scores?
Pros• Statistics are powerful and
illustrative.• Statistics can be applied
consistently.• Statistics can provide you
with a “targeted action” approach.
![Page 14: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/14.jpg)
15
Q5 - AnswerBased on your experience, what are pro’s and con’s
to using statistics to invalidate scores?
Cons• Staying ahead of breaches to
your program is challenging.• Statistics are not a “silver
bullet.”• Exam security requires a
comprehensive plan that is constantly evolving.
![Page 15: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/15.jpg)
16
Q6
Describe a time when you were successful in using statistics to invalidate a test score (or scores). What factors contributed to yoursuccess? Have you ever used only statistics to successfully invalidate a score (i.e., there was no physical or other corroborating evidence of cheating besides the statistical results)?
![Page 16: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/16.jpg)
17
Q6 - AnswerDescribe a time when you were successful in using
statistics to invalidate a test score.
• Studied similarity data, performed analysis
• Reviewed seating charts• Identified index values >7.5• Created a report and referred
to board
![Page 17: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/17.jpg)
18
Q7
Describe a time when you had statistical evidence of aberration, but were unsuccessful in using that evidence to invalidate the score (e.g., you lost a legal dispute or you decided not to pursue the case further for some reason). Explain why you were not successful.
![Page 18: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/18.jpg)
19
Q7 - AnswerDescribe a time when you had statistical evidence of
aberration, but were unsuccessful in using that evidence to invalidate the score. What happened?
• Lack of corroborating evidence
• Unable to convey the compelling nature of the statistical evidence
![Page 19: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/19.jpg)
20
Q8
If a program has a solid testing agreement in place with proctors and candidates, how much time, effort, and money do you estimate is required to invalidate one test score based on statistics?
![Page 20: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/20.jpg)
21
Q8 - AnswerIf a program has a solid testing agreement in place with proctors
and candidates, how much time, effort, and money do you estimate is required to invalidate one test score based on statistics?
This varies based on your organization and processes followed.
• Two examples from our panelists• Example 1: Three primary areas of
expense• Example 2: Internal, automated
process
![Page 21: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/21.jpg)
22
Q9
What internal challenges and obstacles did you confront in instituting your invalidation program? Were they operational? Legal? Political? Please describe.
![Page 22: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/22.jpg)
23
Q9 - AnswerWhat internal challenges and obstacles did you confront in instituting your invalidation program? Were they operational? Legal? Political? Please describe.
• Political – Invalidated scores belonged to customers and partners. Stakeholders had to know what to expect and why this was beneficial.
• Legal – Is process legally defensible? Do we have all the right documentation in place?
• Operational – Continually evolving and staying on top of technologies is key.
![Page 23: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/23.jpg)
26
Q10
Have you instituted any KPIs or other measurements around your use of statistical analyses and invalidations to measure program impact and success? What are they?
![Page 24: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/24.jpg)
27
Q10 - AnswerHave you instituted any KPIs or other measurements around your use of statistical analyses and invalidations to measure
program impact and success? • Metrics re: cases opened, closed,
cancel rates• Metrics re: resolution options
selected by examinee• KPIs to measure effectiveness of
program
![Page 25: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/25.jpg)
28
Q11
What legal agreements or legal foundation need to be in place in order to consider invalidating based on statistics?
![Page 26: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/26.jpg)
29
Q11 - AnswerWhat legal agreements or legal foundation need to be in place in order to consider invalidating based on statistics?
• Clear, comprehensive, signed candidate agreements
• Ethics policies in place• Have a plan, be
comprehensive• Court will look to your agreements
![Page 27: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/27.jpg)
30
Audience Questions
![Page 28: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/28.jpg)
31
Audience Questions
“We see a lot of exam takers who score very high in very short amounts of time. We know that it would be impossible to even read the questions in the short time spans, but it’s hard to prove cheating.
Is there a guideline on the amount of time taken per question that we can use to take these cheaters down?”
![Page 29: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/29.jpg)
32
Audience Question - Answer
• Some reading rates exceed 1,000 wpm (trained speed reading)
• Average adult reading rates are 250-300 wpm on non-technical content
• Reading rates should be verified• High-speed, erratic reading
strongly indicates braindump usage
Is there a guideline on the amount of time taken per question that we can use to take these cheaters down?
![Page 30: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/30.jpg)
33
Conclusions
“I would urge us to reframe our concerns about test data integrity not as cheating concerns, but as a validity issue.”
“I would first strongly caution everyone NOT to do any analyses at all until first developing and adopting policies and procedures about what to DO with the results of any analyses.”
– Gregory Cizek, NCSA, 2013
http://www.caveon.com/tilsa-test-security-guidebook-next-steps/
![Page 31: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/31.jpg)
34
Conclusions“Perhaps the worst situation one could be in would be a situation where analyses have been conducted and it must be admitted publically that nothing has been done with the results.”
“It is important to treat each similarly situated case the same way, and a coherent, comprehensive set of policies and procedures, uniformly applied, is essential.”
– Gregory Cizek, NCSA, 2013
http://www.caveon.com/tilsa-test-security-guidebook-next-steps/
![Page 32: Caveon Webinar Series - Lessons Learned from Using Statistics to Invalidate Scores June 2014](https://reader038.fdocuments.us/reader038/viewer/2022110121/558fc2891a28abd2668b4746/html5/thumbnails/32.jpg)
35
Thank you!
- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”
Liz Burns, Juniper NetworksRachel Schoenig, ACT
Jennifer Semko, Baker & McKenzieDr. Jack Terry, NBEO
Dennis Maynes, Caveon Test Security