Caveon Webinar Series

“The Endless Security Summer: Testing Conference Lessons”

August 22, 2012

2

“The Endless Security Summer”

Steve Addicott• Vice President• Caveon Test Security

John Fremer, Ph.D.• President• Caveon Consulting Services

Lou Woodruff, Ph.D.• Senior Security Consultant, Caveon• Former President, National College Testing Association

3

Agenda

• Conference on Statistical Detection of Potential Test Fraud, May 23-24

• National Conference on Student Assessment, June 26-29

• International Test Commission, July 3-5• Dutch Exam Society, July 6• National College Testing Association, August 1-4• Closing Thoughts

4

Poll Question #1

5

Conference on Statistical Detection of Potential Test Fraud

• University of Kansas’ CETE; Lawrence, KS• First-ever Data Forensics Conference• Gathering of measurement experts &

researchers– Certification– Education– Vendors– Test Publishers– Academics

6

Conference on Statistical Detection of Potential Test Fraud

• Most important takeaway

• Second most important take away

Protecting our tests grows ever harder, and ever more important

THIS CONFERENCE EXISTS!!

7

National Conference on Student Assessment (NCSA)

• June 26-29 in Minneapolis, MN

• Council of Chief State School Officers (CCSSO)

• 2012 Theme: Teaching, Learning, and Assessment through the Next Decade

8

Top Three NCSA Key Takeaways

9

NCSA Key Takeaways

#3—Bombshell! CCSSO is partnering with Center for

Assessment to “re-focus” the conference– Reduced attendance?– Budgets?– Changing priorities?

10

NCSA Key Takeaways

#2—A number of dynamics in state assessments impacting security

– Teacher Accountability• Legislatures passing laws• Using existing assessments to now grade teaching performance

– Assessment Consortia

– Technology• Classrooms• Administration• Learning• Assessments

11

#1 NCSA Takeaway• Despite all these changes,

THERE WAS NOT ONE SESSION FOCUSED ON TEST SECURITY.

NOT ONE.

12

Topics That Could Have Covered…• Thoughtful piloting of security strategies

• By 2014-2015, all state assessments will be CBT– Colossal logistical challenge– Too few PCs and test centers

• Blending individual test programs – Dozens into two– Exams will be shared across states and time zones– Administration windows will last weeks– More technology, more sharing?

13

International Test Commission

• July 3 – 5, 2012

• Group representing testing organizations and national agencies world wide

• Amsterdam, The Netherlands

14

ITC – Major Takeaways

• Growing recognition throughout the world of the importance of protecting IP and preventing cheating

• Attention to standards – ATP, ITC, ANSI, Caveon

• More attention to how to use data to detect misbehavior

• “Trust but verify”

15

CITO & Caveon Session

• CITO – very major provider of testing services to Europe and many non-European countries

• Collaborating with other Dutch testing agencies to promote security standards

• Caveon – evidence of importance of security– Internationalization of testing– Challenges of technologically savvy cheats– Notorious US cases, especially Atlanta PS

16

Faking and Test Security

• Steve Addicott Contribution• Dealing with Media

– Be prepared– Draw on all parts of your organization –

communications, legal, membership, etc.– Have a spokesman– Tell the story from your perspective– Don’t expect the story to go away

17

Dutch Exam Society (NVE) Security Summit

• NVE has 400+ members from dozens of test programs in The Netherlands

• Eight leaders gathered to:– Address Test Security concerns– Consider adoption of Security Standards for

members

• Caveon invited to play a supporting role

18

Top NVE Security Summit Takeaways#3—Times are changing

– Test Security was not, but is now, a front-burner issue

#2—Security Risks are global

#1—Rising tide lifts all boats– Organizations which are considered

“competitive” can collaborate and create positive change

19

Poll Question #2

20

National College Testing Association

11th Annual Conference

Minneapolis 2012

ncta-testing.org

21

College testing administrators and staff

representing over 600 colleges

More than 50 companies that do business

in the testing field

ncta-testing.org

22

Consortium of College Testing Centers

Listing of hundreds of college testing centers

offering proctoring services for individuals

needing to take a secure, proctored examination

Access the listing for free on the NCTA website

ncta-testing.org

23

Six Sessions Devoted to Test Security:

The Keynote Address

The General Session

Four Breakout Sessions

24

Keynote Address

Mark Poole

Product Strategist and Security Expert

Pearson VUE

“The Future of Secure Testing in a Digital World”

25

Highlighted five key trends that are

shaping the next decade of testing:

o a global cheating culture

o high tech test fraud

o testing anywhere securely

o candidate authentication

o data forensics

26

“Demystifying Data Forensics: An Overview of the Logic Underlying Cheating Detection Techniques”

Jim Wollack, Ph.D.Director, Testing and Evaluation ServicesUniversity of Wisconsin, Madison

Presented the basic principles of applied data forensics in a way that everyone could understand.

Reviewed some forensics approaches to identify examinees who copy answers, had prior access to test content, or hired surrogates to test for them

Breakout Session:

27

“Gimme Shelter: Weathering the Media Storm of a Security Breach”

Steve AddicottVice President, Client ServicesCaveon Test Security

Ray NicosiaExecutive Director of the Office of Testing IntegrityEducational Testing Services (ETS)

Faisel AlamTest Security SpecialistLaw School Admission Council (LSAC)

Breakout Session:

28

This panel of security experts shared key aspects of media strategy for testing programs and techniquesfor positioning incidents to various audiences

High profile security breaches were reviewed, highlighting details of what went well and what did not

The use and effectiveness of various media tools was analyzed, and a checklist of media “Do’s and Don'ts” was presented

29

“Security Strategies for Every Stage of the Testing Process”

DeDe Hedlund Jeff PlaceCreighton University Questionmark

Looked at best practices for colleges to protect test content and minimize cheating

Demonstrated the potential for using mobile test centers with examinees testing on tablets such as the Apple iPad

Breakout Session:

30

“Innovations in Online Proctoring”

Rick Beaudry Denis GravelCEO Director, Workforce Development B Virtual Inc. School of Applied Technology

Presented a proctoring service that proctored test takers in any location via web-cam and microphone, with a proctor monitoring the session remotely

Breakout Session:

31

“Detection and Prevention of Test Fraud

The State of the Art”

John FremerPresidentCaveon Consulting Services

Lou WoodruffSenior Security ConsultantCaveon Consulting Services

General Session

32

o Perspectives on the Perpetrators

o Test Security Anatomy

o Contemporary Test Security Responses

o The Technology of Detection

o Large Scale Security Investigations

o Future Test Security Challenges for Higher Ed

o Implications for College Testing Professionals

33

Survey data on the prevalence of cheating among the nation’s youth was shared which showed an alarming level of cheating

Cheating has grown from a strategy for increasing test scores to the manifestation of a growing social norm that accepts dishonesty in many areas of life

The cheating/detection arms race will continue with advances in biometric, data forensic and test delivery techniques enhancing prevention and detection

34

Result of Using Data Forensics in a State Assessment Program

Data Forensic Analyses of school districts statewide

Decisive actions taken on the forensic results

Two years of follow-up test administration monitoring

Found a 72% reduction in test security irregularities

Reduction in "Irregularities"

0.0%

1.0%

2.0%

3.0%

4.0%

5.0%

6.0%

Rep

ort 1

Rep

ort 2

Rep

ort 3

Rep

ort 4

Rep

ort 5

Rep

ort 6

Rep

ort 7

Rep

ort 8

Rep

ort 9

35

The Atlanta Public Schools

Test Security

The Perfect

Storm

36

Test Security

The Perfect

Storm

Test Security

The Perfect

Storm

Mandated assessmentstied to federal funding

Schools and teachers evaluated on test scores

Administrators feeling unjustly treated

Test security left in the hands of the teachers and the principals

37

Governor brought in the GA Bureau of Investigation

Sixty-one agents on the case with subpoena power

District Superintendent, all Deputy Superintendents, and many Principals and Teachers were removed

Who do you trust? Sometimes – no one!

Atlanta Public Schools Test Security Case

38

Take-Aways from the NCTA Conference

College testing professionals are highly aware of and strongly interested in test security

NCTA is a valuable platform for promoting security related initiatives and partnerships between colleges the testing industry

Colleges are and will remain a rich resource for implementing cheating detection and preventionstrategies

39

Questions?

40

Today’s Presenters

Steve Addicott steve.addicott@caveon.com John Fremer john.fremer@caveon.com

Lou Woodruff LCWoodruff@aol.com