www.caveon.com1

The What, Why and How of On-Site Monitoring

March 1, 2017 12:00 p.m. ET

Caveon Webinar Series

Today’s Presenters

www.caveon.com2

Steve AddicottVice President , Sales

Marc WeinsteinVice President, Caveon

Investigative Services

Today’s Questions

•What is on-site monitoring?•Why implement on-site monitoring?•How do we get started with on-site monitoring?•What do we do with the information from monitoring?

www.caveon.com3

On-Site Monitoring

www.caveon.com4

What Is On-site Monitoring?

Trained monitors visit schools and testing rooms during assessment administrations.Monitors will:Review and document school-level test security compliance practices and procedures.Observe test administrators, proctors and students.Review all parts of the testing day.Capture observations using pre-defined checklists for later review.

www.caveon.com5

Why Have A Monitoring Program?• Support the validity of assessment results.• Provide quality assurance for your testing program.– Gather information about implementation and efficacy of training programs.– Measure compliance with test administration processes and protocols.– Identify areas for improvement of process/policies.

• Comply with USED Peer Review Requirements.

www.caveon.com6

What Happens During On-site Monitoring?

www.caveon.com7

Should We Announce Monitoring?•Announced– Pro: Administrators are more likely to ensure compliance with policies and procedures.– Con: School personnel on “best behavior,” so not a true indicator or what happens during assessments that are not monitored.

www.caveon.com8

Should We Announce Monitoring?•Unannounced– Pro: Provides a true indicator of what happens during testing and encourages increased compliance at all times– Con: Added stress for test administrators

www.caveon.com9

During On-Site Monitoring

Look at all parts of the testing day–Training logs and proctor agreements.–School-level secure testing materials management.–Proctor and student login procedures.–Actual testing sessions in testing rooms.–Final sign-off/ secure testing materials management.

www.caveon.com10

During On-Site Monitoring

Focus on process and people– Test administration process.– School staff/proctor roles and behaviors.– Student behaviors.– Testing environment (classroom, computer lab).

www.caveon.com11

During On-Site Monitoring•Use objective checklist or form– Yes/no (binary) checklist items to eliminate discretion of monitors and allow for data analytics across schools.– Require explicit narrative descriptions of any observed potential irregularity or deviation from policies.– Clear procedures for data capture and management.– Archiving and evidence preservation.

www.caveon.com12

During On-Site Monitoring•Consider electronic tools for incident management– Real-time data capture.– Data analytics across schools.– Workflow tools for incident management.– Archiving and evidence preservation.

www.caveon.com13

What to do after On-Site Monitoring

www.caveon.com14

Use of Monitoring Data and Results•Do irregularities or potential violations observed during monitoring require follow up investigation?•What level of monitoring data and results will be shared with schools and staff?•Should public reports of monitoring be released?

www.caveon.com15

Use of Monitoring Data and Results• Evaluate with other test security data sources.–Incident or irregularity reports.–Data Forensics or other test results analysis.•Compare monitoring data points across schools.• Evaluate monitoring data within overall test security context.

www.caveon.com16

Take Action• Provide feedback to staff at observed sites.•Conduct follow up investigations if warranted.•Select sites for future monitoring.• Improve test materials, instructions, or training.

www.caveon.com17

Getting Started

www.caveon.com18

Three questions to ask•What are the goals of monitoring?– Quality assurance.– Ensuring compliance.– Evaluating training effectiveness.•What resources are available for monitoring?•What will we do with the data from the monitoring?

www.caveon.com19

Resource Document

www.caveon.com20

Six Steps

1. Set purpose and goals.2. Determine budget/funding source.3. Select sites.4. Evaluate staffing options.5. Develop audit forms.6. Set periodic program reviews.

www.caveon.com21

Other Important Things to Consider•Communication plans to stakeholders.• Post test surveys of stakeholders about testing experience.• Staffing.

•Case Study: Messaging

www.caveon.com22