2011 NATIONAL CONFERENCE ON STUDENT ASSESSMENT (NCSA)

THE PEABODY ORLANDOORLANDO, FLORIDA

JUNE 22, 2011

Alignment Issues between Computer Adaptive Tests and

Academic Curriculum Standards

NORMAN L. WEBBWISCONSIN CENTER FOR EDUCATION

RESEARCH

Current Requirements

States need to produce evidence to show that its assessment system captures the full range and depth of knowledge and skills defined in the state standards.

Main Questions for Judging Alignment

Are students held accountable. . .1.for all of the main academic content standards?

(category)

2. to perform at a level of complexity comparable to what is

expected by the academic content standards? (complexity)

3.for the full range of content with in each of the main academic content standards? (range)

4.for the same degree of emphasis on topics and subtopics as

included in the main academic content standards? (balance)

Five Technical Components of CAT

1. Calibrated item pool2. Starting point on entry

level3. Item selection algorithm4. Scoring procedure5. Termination criterion

Main Issues for a CAT to be Considered Aligned with Academic Content

Standards

● Item selection algorithm ● Termination criterion

Item Complexity and Item Difficulty

Complexity

High Low

Difficulty

High

Sophisticated content and challenging processes

Repetitive parts(e.g. recall 10,000 digits of PI)

Low

Sophisticated content with deep instruction(e.g. emphasis on problem solving)

Memorize algorithm to answer well define question(e.g. divide four digits number by two digits number)

Convergence on Ability and Range

Increased efficiency in ascertaining a student’s ability or proficiency level can be counter to assessing a student on the full range of content within a set of academic content standards.

Possible Strategies for Analyzing Alignment Between a CAT and Academic Content

Standards

Analyze the full calibrated item pool

Analyze a random sample of items from the calibrated item pool

Analyze Item Selection Algorithm and Termination Criterion to evaluate if category, complexity, range, and

balance are considered

Strategies (continued)

Analyze pathways and collection of items used to determine a student’s proficiency1.Select pathways used to identify students at different levels of proficiency (e.g. high proficiency, at proficiency, below proficiency)

2.Select pathways and collection of items at random

3.Select most common pathways and collection of items

Strategies (continued)

Conduct a correlation study of CAT proficiency scores compared with scores on an assessment judged to be fully aligned with the academic content standards

Others?