1. Hashtag #HStakesCaveon Webinar SeriesPreventing and Detecting Cheating on High Stakes TestsPresented by:John Fremer, President, Caveon Consulting ServicesVickie Cartwright, Sr. Director, Accountability, Research, andAssessment, Orange County School District

2. Presentation Overview Overall Summary External Assistance Training Security Audit Data Forensics Investigations Dealing with Media 3. Overall Summary Provide Appropriate Training Evaluate the Adequacy of YourSecurity Procedures Perform Cheating Analyses Routinely Follow Up Thoroughly on AllSuspicions Take Appropriate and Public Action 4. Providing Appropriate Training Focus on Test Administration Greatest vulnerability for all programs Biggest bang for the buck Build on State and Vendor Materials Make Sure Everyone is Trained Have Building AdministratorsParticipate 5. Issues for which ExternalAssistance is Recommended External Test Security Audit Data Forensics Web Patrol (Monitoring Internet) Security Investigations 6. Test Security Audit: The Importance of Knowing Validates your Security Practices Identifies Vulnerabilities Recommends and Prioritizes Actions toCorrect Shows Your Seriousness about Security 7. Test Security Audit: What is it? Systematic Review by ExperiencedProfessionals Objective Third Party Evaluation Use Formal Standards Seek Practical and Specific Advice 8. Need External Security Auditors Do Your Own Security Reviews Regularly? Have Such Reviews as an Assignment for aKey Staff Member Cant do a Full Security Review Internally Its like evaluating your own children Are they more beautiful or talented or smart? Need objective and well-informed perspective 9. Caveon Test Security Audit Topics1. Security Plan 8. Test Administration2. Roles & Responsibilities9. Test Scores & Results3. Budget & Funding10. Physical security4. Legal Precautions & 11. Information SecurityAgreements 12. Web & Media Monitoring5. Test & Item Design 13. Security Awareness &6. Test & Item Development & TrainingMaintenance 14. Security Incidence7. Test PublicationResponse Plan 15. Conducting Security Investigations [New] 10. Questions? 11. Psychometric Analyses -Detecting CheatingData Forensics Looking at Your Test Data Has a Test been Compromised? When and Where was it Compromised?Traditional Procedures No Longer Good Enough 12. Data Forensics School Districts Type of Data Available to DistrictsUsually Limited Gains Analyses Erasure Analyses in Paper and Pencil - byInspection Not practical except when there is a tip or something observed More Sophisticated Analyses RequireMore Data and are Very Desirable 13. Data Forensics Benefits Use Data to make Informed Decisions Train from Known Threats Detect and Deter Cheaters Stay Ahead of Cheating Methods Evaluate Test Admin SecurityProcedures Evidence of Strong Action Bringing inIndependent Experts 14. Other Data Forensics Analyses Similarity of Responses for Pairs andGroups of Students Aberrance Failure to Have a LogicalPattern, e.g., Answering More HardItems Correctly than Easy Ones Complete Erasure or Answer ChangingAnalyses What % Wrong to Right 15. Success Story - State AssessmentReduction in "Irregularities" 72% reduction 2 years6.0% Test admin monitoring5.0% Conditional status 4.0% Districts follow through 3.0% Students confess 2.0% Coordinator training 1.0% 0.0%Report 1 Report 2Report 3 Report 4Report 5 Report 6Report 7 Report 8Report 9 16. Web Patrol Are Secure Items on the Internet? Occurs in High Stakes Testing Long Testing Windows Tend to Lead toLeakage (Here comes the consortia) Involves Searching Tools, ExpertSleuthing, and Quick Follow-Up Very Common with CertificationPrograms 17. Any Questions? 18. Atlanta Public Schools Story Persistent Accusations of Cheating inAPS Schools Consistent Denials Reports are wrong Newspaper is prejudiced Data Revealed Unmistakable Evidenceof Improprieties Extreme cases very pronounced oneover 10 to 52nd power 19. Media Lessons from High Profile Incidents Expect Media Attention Get Out Front with Your Own Story Cultivate Local Media Most do not understand test results Looking for interesting story a no problem administration is not a good story Learn Who You Can Trust Dont Expect a Cheating Story to Go Away You Cannot Control It 20. FURTHER INFORMATION Slides & recording available at:www.caveon.com/resources/webinars Caveon Security Insights Blog http://www.caveon.com/blog/ Twitter @Caveon LinkedIn Caveon Company Page Caveon Test Security Group Please contribute!