ESSA Assessment

Peer Review

Craig Walker, Executive Director of State Assessment

Oklahoma State Department of Education

1

Central Tenets of Assessment Peer Review

• Support States in developing and administering assessment systems that provide valid and reliable information on how well students are achieving a State’s challenging academic standards to prepare all students for success in college and careers in the 21st century.

• The Standards for Educational and Psychological Testing, nationally recognized professional and technical standards for educational assessment, were updated in Summer 2014.• Increased attention to fairness and accessibility and the expanding role of

technology in developing and administering assessments.

2

Peer Review: Critical Elements

The six sections of critical elements of assessment peer review:

(1) Statewide System of Standards and Assessments,

(2) Assessment System Operations,

(3) Technical Quality – Validity,

(4) Technical Quality – Other,

(5) Inclusion of All Students, and

(6) Academic Achievement Standards and Reporting.

3

Ass

essm

ent

Syst

em

Op

erat

ion

s

Test design/Dev.

Item Dev.

Test Admin.

Monitoring

Test Security

Data Privacy/Integrity

4

Peer Review Outcomes• Meets Requirements

• No further action is required of a state unless the state makes significant changes to its standards or assessments

• Substantially Meets Requirements• State meets most of the components. The Department expects that a State should

be able to provide the additional information within one year.

• Partially Meets Requirements• State does not meet a number of the requirements. Condition placed on certain

federal funds & Quarterly conference calls to monitor a state’s plan to correct deficiencies.

• Does Not Meet Requirements• A component of the State’s assessment system does not meet most of the

requirements. Cease Administration? Condition placed on federal funds. Possible designation of high risk status and/or Compliance Agreement.

5

Evidence-Based Peer Review

6

What is Evidenced-Based Peer Review?

According to the ED, “The Department’s assessment peer review focuses on the processes for assessment development employed by the State and the resulting evidence that confirms the technical quality of the State’s assessment system.”

• To determine if a State has met ESEA standards and assessment requirements, the Department uses a peer review process involving experts in the field of educational standards and assessments.

7

What Evidenced-Based Peer Review IS NOT

• Providing only statute, administrative code, policies, manuals for specific components

• Themes for not meeting requirements center around the lack of evidence, including:• Training Sign-in sheets for school district personnel for state-led test

administration/program training

• Actual letters calling out compliance issues, investigative notes, and corrective action letters

• Summaries or reports addressing outcomes of monitoring

8

SEA Preparation• Planning meeting with staff

• Assign staff to analyze the evidence that each division within an agency possesses• Break down the critical elements and explicitly assign

• Brief other divisions on evidentiary pieces needed for peer review

• Create a repository system to easily categorize evidence using the critical elements map

9

Repository System Example

10

2.3 Test Administration

11

2.3 Test Administration: ED Guidance

• The State implements policies and procedures for standardized test administration• Has established and communicates to educators clear, thorough and

consistent standardized procedures for the administration of its assessments, including administration with accommodations

12

Evidence Examples

• Test Administrations Manuals

• District Test Coordinator Trainings (required for each district)• Sign-in sheets from SEA training sessions

• Presentation Slides

• SEA provided training modules for Test Administrators and Test Proctors• Certificates of Training

• Accommodation Manuals

13

Administration with Accommodations

14

2.4 Monitoring Test Administration&

2.5 Test Security

15

2.4 Monitoring Test Admin: ED Guidance

The State adequately monitors the administration of its State assessments to ensure that standardized test administration procedures are implemented with fidelity across districts and schools.

16

2.4 Monitoring: Peer Review Feedback

• Evidence that the State monitors the administration of its State assessments to ensure the fidelity of test administration procedures.

• Evidence that the State applies the test monitoring procedures for its general assessment test administration to the alternate assessments.

• Evidence that specifies which assessments are to be monitored.• Evidence of test monitoring procedures • Evidence of the State’s monitoring process:

• Selecting which districts and schools will be monitored by the State educational agency staff.

• The cycle for monitoring schools and districts across the State; Schedules for monitoring;

• Monitors’ roles and responsibilities of key personnel • Summary of the results of the State’s monitoring

17

2.5 Test Security: ED Guidance• The State has implemented and documented an appropriate set of policies

and procedures to prevent test irregularities and ensure the integrity of test results through:

• Prevention of any assessment irregularities: • Maintaining the security of test materials, • Proper test preparation guidelines and administration procedures, • Incident reporting procedures, • LEA consequences, and • Annual training requirements

• Detection of test irregularities

• Investigation of alleged or factual test irregularities.

• Remediation following any test security incidents

18

2.5 Test Security: Peer Review Feedback

• Documentation of a process to prevent, detect, report, investigate, and remediate assessment irregularities.

• Documentation of implemented policies and procedures to address test irregularities, and documentation that the policies have been implemented (e.g., summaries of reported allegations and steps the State took to investigate reports of allegations).

• Evidence of consequences in the State for confirmed violations of test security.

• Evidence of annual training requirements for test security policies and procedures for educators.

19

Evidence• Statute or Administrative code authorizing the state to monitor

assessments

• Policies surrounding test security & assessment monitoring, especially the process used to address irregularities

• Desk Monitoring & On-site monitoring protocols & checklists• Compliance/non-compliance letters submitted to districts

• Citations • Corrective Actions/outcomes• Impact on overall protocol for administering state assessments

• Program cycles and methodology to select districts/sites• Observational Data• Statistical Analysis

20

Evidence• Chain of Custody• Organizational policies and practices to maintain test security

• How to properly transmit items between vendor and SEA/assessment staff

• How to maintain test/item security during item reviews, standard setting, and other external committee meetings that may expose items.

• What’s the process and mode of reporting test security issues

• Technology-specific issues• Post administration data forensics

• Person fit analyses

• Summary of Test Security Incidents• Contingency Plans for addressing large-scale breaches

21

Evidence

• Evidence of procedures for remediation of test irregularities includes documents such as:

• Contingency plan that demonstrates that the State has a plan for how to respond to test security incidents and that addresses: • Different types of possible test security incidents (e.g., human, physical,

electronic, or internet-related), including those that require immediate action (e.g., items exposed on-line during the testing window);

• Policies and procedures the State would use to address different types of test security incidents (e.g., continue vs. stop testing, retesting, replacing existing forms or items, excluding items from scoring, invalidating results); o

• Communication strategies for communicating with districts, schools and others, as appropriate, for addressing active events.

22

Examples

• Special Investigation• District personnel alerted SDE that a major violation was observed involving

mass erasures.

• Snap Chat• District self reported that a student snap chatted during the state test

• Erasure Party• District reported teacher made copies of the actual test to create “study

guides”

23

Policies & Procedures

Training

Implementation

& ExecutionData Collection

Analysis of observational &

statistical

24