Computer Adaptive Testing

FAIRFAX COUNTY PUBLIC SCHOOLS

A JOINT PRESENTATION BY:

THE OFFICE OF INSTRUCTIONAL SERVICES

(PRE K – 12 CURRICULUM & INSTRUCTION)

AND

THE OFFICE OF STUDENT TESTING

1

VDOE introduces

Computer Adaptive Testing for Grade 6 Math 2

This year, in Virginia, the 6th grade Math SOL will

be administered using a Computer Adaptive

Testing (CAT) model.

The CAT model was piloted during the 2013-2014

school year for both Mathematics and Reading

SOLs.

What is CAT?

3

Briefly, CAT provides a customized assessment for

every student. Answers are scored as each

question or problem is answered, and the

computer selects subsequent items based on the

student’s performance on the items already

completed.

What is CAT?

4

Computer Adapted Testing is a proven technology;

it has been around for over 30 years and has been

shown to increase student engagement and test

security.

For example, since 2002 Virginia has used the

Algebra Readiness Diagnostic Test (ARDT) which

is administered in the CAT format.

The GRE exam is also a CAT.

A Computer Adaptive Test Versus a Traditional Test

• A Computer Adaptive Test is an assessment that

is customized for every student based on how the

student responds to the questions.

• A traditional test is an assessment in which all

students who take a particular version of the test

respond to the same test questions.

5

CAT Implementation Schedule

VDOE’s Implementation Schedule:

It is very likely that the next SOL test or tests that become computer

adaptive will be in mathematics. However, the timeline for

implementation in mathematics, as well as other content areas, is

uncertain due to budget constraints.

6

Similarities between a CAT and a Traditional Online Test

Both test types:

• Assess the Standards of Learning (SOL) for the course (e.g., Grade 6 Mathematics).

− This means that the CAT does NOT assess standards below or above grade level.

− The CAT adjusts the difficulty level of the questions from within the item pool for the test.

− ARDT and GRE are CATs that use a different model and assess across a wider range of difficulty

7

Similarities between a CAT

and a Traditional Online Test

Both test types:

• Contain sample items*, multiple-choice questions,

and approximately 15% technology-enhanced items

(TEIs).

• Are untimed, but should generally be completed

within one school day.

*The responses to the sample items do not impact the difficulty

level of the questions selected for the student in a CAT.

8

Similarities between a CAT and a Traditional Online Test

Both test types:

• Require students to log in to the test using a Student Authorization Ticket.

• Allow students access to the same test manipulatives (e.g., calculator, scratch paper).

• Allow students access to the same online tools on the TestNav tool bar (e.g., highlighter, ruler).

• Contain the same section breaks (if applicable).

• Allow for accommodations.

9

Differences between a CAT and a Traditional Online Test

The number of questions (or test items) on a CAT will

be different from the number of test items on a

traditional test. However, all students taking a CAT

will be administered the same number of items.

See the Grade 6 Mathematics Test Blueprint on the

VDOE web site at, as referenced in Superintendent’s

Memo 251-14.

10

Differences between a CAT and a Traditional Online Test

The Grade 6 Mathematics CAT has 45 operational items and 8 field test items for 53 total items.

The Grade 6 Mathematics traditional test has 50 operational items and 10 field test items for 60 total items.

Grade 6 Mathematics Test Blueprint Summary Table

11

Differences between a CAT and a Traditional Online Test

Difference in item type:

• The CAT test will have some multiple-choice and

technology-enhanced items with only 3 answer

options.

• The directions in the Examiner’s Manual mention

this. However, prior to the day of testing, students

should be made aware so they will not think a

question with three answer options is an item with

an error.

12

12

Differences Continued… Navigating Through the Test

On a CAT:

• Each question must be answered completely

before the student can move to the next question.

• Questions cannot be skipped.

• Students cannot go back and change responses to

questions that have been previously answered;

there is no “Previous” button.

13

On a CAT:

• Since students must answer each question before

moving forward and students are not allowed to go

back to previous questions, it is not necessary to

flag questions for review. Therefore, the Flag for

Review button has been removed.

• Since all questions must be answered to reach the

end of the section or test, there is no Section

Review Screen.

14

Differences Continued… Navigating Through the Test

On a CAT:

• The test does not have a Final Submit button. It is

automatically submitted for the student after the

last question has been answered.

15

Differences Continued… Navigating Through the Test

On a CAT:

• The student cannot submit a test if it has not been

completed.

• All questions must be answered.

16

Differences Continued… Navigating Through the Test

On a CAT:

• Students may take individual breaks, but no group

break is allowed since each student will have a

unique set of test questions.

• There is a transition screen between the

non-calculator section and the calculator section

of the test. Students must be given calculators

before continuing.

17

Differences Continued… Administering the CAT

The Test Directions

instruct students to

check this box once a

calculator is received,

but students will be

able to proceed past

this screen without

entering a seal code

and without checking the box.

18

Differences Continued… Administering the CAT

Administration of a CAT: Student Accommodations

All* student accommodations that are available on

a traditional test are available on a CAT. However,

the manner in which accommodations are

administered may be different.

*Plain English Audio CAT will be available but the

implementation may be delayed until the 2015-

2016 school year.

19

Because each student will have a unique set of test

questions, the Read Aloud or

Interpreting/transliterating accommodation must be:

• Provided in a one-on-one testing environment.

• Administered without an Examiner Authorization ticket. The

Examiner will read from the student’s computer, but the student

must maintain control of the computer. The Examiner is not

able to review the test ahead of time.

With either accommodation:

• The session must be recorded or a proctor must be present

throughout the administration of the test.

Administration of a CAT: Read Aloud or Interpreting/transliterating Accommodation

20

21

Remember that a student receiving the Read Aloud accommodation on classroom assessments may be administered the Audio format of the test on the day of SOL’s.

Please refer to the FCPS SOL Accommodations Reference Chart for further details:

http://fcpsnet.fcps.edu/ssse/SpecialEducation/assessment/sol/SOL-Accommodations.pdf

Administration of a CAT: Read Aloud or Interpreting/transliterating Accommodation

New Special Testing Accommodations Sections

created in test manuals for Computer Adaptive

Tests:

• Read-Aloud Guidelines for Computer Adaptive Tests (CAT)

• Interpreting/Transliterating Computer Adaptive Test (CAT) Items

Located in:

• Appendix B - Examiner’s Manuals

• Appendix C - Test Implementation Manuals (TIM)

22

Administration of a CAT: Read Aloud or Interpreting/transliterating Accommodation

Administration of a CAT: Reporting a Problem During Testing

If a student is completing a CAT and raises a

question or concern regarding a test item, please

contact a Test Administration Specialist with the

following information: school name, the reporting

student’s STI, and the number of the item (e.g., #5,

#16) the student is questioning. This is necessary

to identify the specific test item the student is

seeing.

The Office of Student Testing will communicate with

VDOE regarding the student’s concern.

23

23

Scores and Score Reports

CAT Scaled Scores:

• Scaled scores range from 0 to 600, with 400

indicating pass/proficient and 500 indicating

pass/advanced.

• The scaled score is determined by the number of

questions answered correctly AND the difficulty

level of the questions answered correctly.

• There is no longer a cut score for passing, and the

raw scores will no longer be reported.

24

Grade 6 Mathematics Scaled Scores:

• Scaled scores have the same meaning across all

grade 6 mathematics tests that assess the 2009

Mathematics Standards of Learning.

• This means, for example, that a scaled score of

400 on this year’s CAT test has the same meaning

as a score of 400 on last year’s traditional grade 6

mathematics online test or the grade 6

mathematics paper test.

Scores and Score Reports

25

CAT individual score report availability:

• Individual score reports will be available within 24

hours after testing is completed.

Scores and Score Reports

26

For a CAT, score reports will include:

• The student’s score, performance level, and a

description of the student’s performance level.

• Item level performance information for the student

through the Student Details by Question (SDBQ)

report.

Scores and Score Reports

27

Superintendent’s Memo 255-14 announced these

resources on the VDOE web site :

• A video entitled A Guide to Computer Adaptive

Testing;

• A video entitled Computer Adaptive Testing in

Virginia;

• A document titled Frequently Asked Questions

About Computer Adaptive Testing.

CAT Resources

28

Additional resources on the VDOE website include:

• A “Traditional Test versus CAT” comparison

document.

• A CAT Training Test.

CAT Resources

29

• The CAT Training Test provides students with a

simulated experience for navigation through a CAT form.

• Students can practice answering a small number of

questions (grade 6 mathematics released items) to

familiarize themselves with CAT features that are similar

to a traditional test and those that are new or different.

• The CAT Training Test will be shorter than an actual CAT

(due to a limited number of grade 6 mathematics

released test items) and will not have the same level of

customization for students. It serves as a brief

simulation for CAT training purposes only.

The CAT Training Test

30

CAT Questions?

31

Questions may be sent to either a Math Specialist or

a Test Administration Specialist

Rose Moore Coordinator, Mathematics (K-12 Instruction) 571-423-4725

David Van Vleet Education Specialist- MS Mathematics 571-423-4723

Spencer Jamison Education Specialist- ES Mathematics 571-423-4728

Sheri Masich Test Administration Specialist 571-423-1403

Erica Meadows Test Administration Specialist 571-423-1419

Stuart Pierce Test Administration Specialist 571-423-1413