Lessons Learned fromThe European Association of Test Publishers Conference and

The Conference on Statistical Detection of Potential Test Fraud

November 20, 2013

Caveon Webinar Series:

John Fremer, President Caveon Consulting ServicesSteve Addicott, Vice President, Client Services

Dennis Maynes, Chief Scientist, Caveon Data Forensics

Agenda for Today

John Fremer, Observations from E-ATP Handbook of Test Security CCSSO TILSA Test Security Guidebook The Emergence of Social Media

Steve Addicott, Observations from E-ATP and CSDPTF ATP/CCSSO OBP ITC Guidelines Electronic Badging ATP Security Committee CSDPTF Observations

Dennis Maynes, Observations from CSDPTF Progress, Research and Direction Techniques that work, and don’t work Challenges in this field

Lessons Learned fromThe European Association of Test

Publishers Conference

John Fremer

5th Annual E-ATP conference: Growing Talent In Europe: Gaining Advantage Through Assessment

Brings together European and other international test publishers and related organizations

Held in beautiful St. Julian, Malta

European Association of Test Publishers (E-ATP) Annual Conference

1. What security vulnerabilities exist for all genres of testing?

2. The critical importance of security planning

3. Practical and proven strategies for preventing and detecting cheating

4. How security breaches have been dealt with in specific programs

5. What lessons have we learned from past instances of testing misbehavior?

HANDBOOK OF TEST SECURITY, FIVE TAKEAWAYS

In many high stakes testing programs:1. Cheating detection statistical analyses

will be performed routinely2. Computer-based testing will increasingly

become the norm3. Technology developments will be critically

important to test security4. The internationalization of the program

will increase5. “Test Security Expert” will become a

recognized and valued position

HANDBOOK OF TEST SECURITY,FIVE PREDICTIONS

Review the Table of Contents “Skim read“ the chapters Earmark chapters and sections for staff to

read or for you to return to Read sections as you deal with the issues

addressed

ADVICE FOR THE READER

To get the Handbook,http://www.psypress.com/books/details/9780415816540/ use discount code HYJ82 for a 20% discount.

TILSA GUIDEBOOK

Guidebook: Preventing, Detecting, and Following up on Testing Irregularities

Center for Chief State School Officers

Designed for State Assessment Directors

Communicate Zero Tolerance1. Clear 2. Consistent3. Frequently emphasized

messages Develop a Security Plan

Comprehensive Up to date

PREVENTION(Sample)

Use Multiple Data Forensics Methods1. Unusual Gains2. Similarity3. Erasures (Answer Changing)4. Person Fit Analyses

DETECTION(Sample)

Areas Needing Attention Level of evidence warranting an

investigation Roles for state, district, and other

staff Time lines Model investigations kit

TEST SECURITY INVESTIGATIONS (Sample #1)

Model Investigations Kit Clear definition of

responsibilities Procedures for evaluating

evidence Planning and conducting

interviews Developing a report with

recommendations

TEST SECURITY INVESTIGATIONS (cont.)(Sample #2)

To get a copy of the Guidebook,

Download a PDF for free or purchase your own copy for only $3.59.

http://www.ccsso.org/Resources/Publications

TILSA GUIDEBOOK

SOCIAL MEDIA

Importance of Social Media -- several presentations

Sessions suggest using “Glocal” approach

Social Media continue to gain users in testing internationally

Ignite Session: Are You One Post Away From a Social Media Crisis?

Speaker: Matthew Poyiadgi, Managing Director, Pearson VUE

3 P’s to deal with a social media crisis: Police Proactive Stance Proportional Response

Customers expect a 1-day response from you. 20% get it.

Ignite Session: Are You One Post Away From a Social Media Crisis? (cont)

Summary Choose your communication channels wisely Write your guidelines – make them practical Choose and train those representing your

agency Be proactive – put your guidelines in action Be proportionate with resources and

responses Track what you do – watch for patterns

Be personable on social media! You’re dealing with people!

Lessons Learned fromThe European ATP Conference &

The Conference on Statistical Detection of Potential Test Fraud

Steve Addicott

ATP/CCSSO Operational Best Practices for Statewide Large Scale Assessment Programs

International Test Commission Guidelines for the Security of Examinations, Tests and Other Assessments

Important Test Security Resources

Published in Spring 2010: Enhance the quality, accuracy,

and timeliness of student test data derived from large scale assessments

Strengthen public confidence in the accuracy and quality of testing data and their uses

ATP/CCSSO Operational Best Practices

2013: Revised OBP for CBT and Online Testing Just published September 2013!

May not be precisely applicable to “…(test programs) used on an international basis”

Provides a solid framework from which others might seek to define a set of practices tailored to their testing programs

ATP and the CCSSO encourage others to use this document

ATP/CCSSO Operational Best Practices

To get a copy of the OBP,

Purchase a copy for $29.99 at

http://www.ccsso.org/resources/publications

ATP/CCSSO Operational Best Practices

International Test Commission Guidelines for the Security of Examinations, Tests and Other Assessments

International Test Commission (ITC) Security Guidelines

With increasing security problems, the guidelines are:

intended to share key elements of security best practices to promote better security, and

defend the value of scores produced through the assessment process

ITC Guidelines: Purpose

All high-stakes assessments, tests and exams

All stakeholders in the assessment process (publishers, users, students, test developers, etc.)

The entire assessment process from development to administration to results processing and scoring

Across all areas of testing (e.g., education, employment, certification/licensure government, clinical psychology, etc.)

ITC Guidelines: Scope

The ITC Guidelines provide answers to these questions:

Threats. What are the dangers?

Test Fraud. How can we deal effectively with increasing security problems?

Terminology. How can we best communicate about test security

ITC Guidelines: Value

Electronic Badging

What is electronic badging?

Big industry players involved

Compelling value

Badging + accreditation + assessment = ???

Stay tuned

ATP Security Committee

Picking up from last February’s conference

Annual Survey results released: https://www.createspace.com/4418924

Hands-on demonstrations of new tech innovations used by test thieves and pirates

Harder and harder for the “good guys” to keep up

About the Conference

The Conference on Statistical Detection of Potential Test Fraud (CSDPTF)

2nd Annual Conference this year

Presents various methods of cheating detection and offers an important forum to examine a variety of methods

Huge thanks to University of Wisconsin at Madison’s Dr. James Wollack and team!

Conference on Statistical Detection of Potential Test Fraud

Last year’s event was groundbreaking

This year, growing momentum… Premier academic & industry organizations

supporting/sponsoring Attendance steadily increasing Publishing of selected papers Plans for future years already underway! Rumors of a scholarly journal – stay tuned!!

Conference on Statistical Detection of Potential Test Fraud

Compelling dynamics behind the conference: Growth in testing Stakes growing ever-higher Focus on trustworthy test results How could we NOT focus on statistical detection?

Conference on Statistical Detection of Potential Test Fraud

Challenge: Current research on the use of statistics. What about the policies and procedures- the legal

defensibility- wrapping their use?

Lessons Learned fromThe Conference on Statistical Detection

of Potential Test Fraud

Dennis Maynes

There were three general sessions on Friday with four or five presentations and a discussant.

About ten poster presentations were on display Friday evening.

There were three concurrent sessions on Saturday with five presentations in the first session, four in the second, and a symposium/workshop in the third.

The conference was attended by over 120 researchers!

Summary of CSDPTF

Similarity and Person Misfit Pre-knowledge and “Braindumps” Unusual Gains Robustness and Sensitivity Issues Test Tampering Computer-Based Testing Policy Test Fraud Detection Challenge Security Planning

Topics Covered at CSDPTF

CSDPTF: 2012 versus 2013

The conference this year was more varied in terms of topics and research than last year. Attendees were from all over the world. The conference has expanded in its reach, focus, and attractiveness.

For this field to progress, the following are needed:

A set of data sets that have known characteristics of security breaches.

Commonly accepted ways to compare the efficacy of methods.

Consensus on the utility and implementation of statistical methods.

Consensus on how to make inferences concerning potential test fraud.

Guidance to practitioners on how to use the statistics.

Progress

The current state of research is that of individual or organization-based efforts with low cross-fertilization between those efforts.

Research

Greg Cizek presented the keynote speech. He discussed where this field has been and where it is going.

Direction

Test Tampering aka “Erasure Analysis”

Similarity aka “Answer Copying”

Techniques that work well and/or are gaining greater acceptance

Detection of item compromise in all its forms such as braindumps, rogue review courses, and disclosure of live exam content by teachers and/or instructors

Analysis of gain scores for detecting potential test compromise

Analysis of response times to detect potential test compromise

Techniques that do not work well

1. Using statistics to invalidate test scores because the scores are not valid. The problem is that the public and measurement professionals have focused on the behavior not the scores.

2. Conducting investigations when the evidence that misbehavior occurred is circumstantial. There are no missing persons, broken doors, or misappropriated funds.

Challenges Practitioners in This Field Face

3. Demonstrating the science behind the techniques that are being used:

Reproducible results Sound reasoning Quantitative measures Defensible assumptions Measurement of error

Challenges Practitioners in This Field Face

Questions?

Caveon Online

Caveon Security Insights Blog http://www.caveon.com/blog/

twitter Follow @Caveon

LinkedIn “Caveon Test Security” Group

Please contribute! Dr. John’s Test Security Tip of the Day Caveon Security Minute

Facebook “Like” us!

Dr. John Fremer

President, Consulting Services

John.fremer@caveon.com

@TestSecurityGuy

Steve Addicott

Vice President, Client Services

Steve.addicott@caveon.com

@SdAddicott

Dennis MaynesChief Scientist

Dennis.Maynes@caveon.com@DennisMaynes

Thank you for your attendance!

At Caveon, we fundamentally believe in quality testing and trustworthy test results.