Issues Related to Judging the Alignment of Curriculum Standards

and AssessmentsNorman L. Webb

Wisconsin Center for Education ResearchUniversity of Wisconsin-Madison

Annual Meeting of the American Educational Research Association

Montreal, April 11, 2005

This work was supported by a subgrant from the U. S. Department of Education (S368A030011) to the State of Oklahoma and a grant from the National Science Foundation, (EHR 0233445) to the University of Wisconsin—Madison. Any opinions, findings, or conclusions are those of the author and do not necessarily reflect the view of the supporting agencies.

Webb Alignment Process

Identify Standards and Assessments Select 6-8 Reviewers (Content Experts) Train Reviewers on DOK Levels Part I: Code DOK Levels of the

Standards/Objectives Part II: Code DOK Levels and

Corresponding Objectives of Assessment Items

Reports

Standards DOK analysis

Degree of alignment by standard

Source of Challenge

Reviewers’ Notes

General Comments Made by Reviewers

Reliability Among Reviewers

Overall Finding of the Degree of Alignment

Data Tables

Acceptable levels on four alignment criteria Source of challenge DOK by item and intraclass correlation Notes DOK level and objective code by item Objectives by item Items for each objective Number of reviewers coding an item by objective

Alignment Criteria

Categorical Concurrence

Number of items per standards

Depth-of-Knowledge Consistency Percent below, at, and above

Range-of-Knowledge Correspondence Percent of objectives with at least one item

Balance of Representation Index 0 to 100

What are appropriate acceptable levels for each criterion?

Number of items for one standard Distribution of items by complexity Number of standard-objectives with at

least one item Distribution of items among objectives

under a standard

Alignment Levels Using the Four Criteria

Alignment Level

Categorical Concurrence

Depth of Knowledge

Range of Knowledge

Balance of Representation

Acceptable 6 items per standard

50% 50% 70%

Weak --- 40%-49% 40%-49% 60%-69%

Unacceptable Fewer than 6 items per standard

Less than 40%

Less than 40%

Less than

60%

State ACategorical Concurrence for Grade 3 Science (N=55 items)

Standards HitsCat. Concurr.

Title Mean S.D.

3.1 - History/Nature 1 0 NO

3.2 - Inquiry 17.38 2.12 YES

3.3 - Unifying Themes 7.5 4 YES

3.4 - Subj Matter/Conc 33.5 1.94 YES

3.5 - Design/Applic 2.12 1.27 NO

3.6 - Personal/Social 4.75 1.09 NO

Total 66.25 5.78

State BDepth-of-Knowledge Consistency High School Mathematics (N=51 items)

Standards

 

# Hits % Under % At % AboveDOK

Consistency 

Title M M M M  

I - Patterns, Relationships and Functions

10.44 83 17 0 NO  

II - Geometry and Measurement 13 20 51 29 YES  

III – Data Analysis and Statistics 13.44 58 40 2 WEAK  

IV - Number Sense and Numeration

2.78 25 61 14 YES  

V - Numerical and Algebraic Operations and Analytical ...

10.67 30 57 12 YES  

VI - Probability and Discrete Mathematics

6.89 42 56 2 YES  

Total 57.22 43 47 11  

State B DOK Consistency

0

20

40

60

80

100

I - P

atte

rns/

Fcns

II - G

eom

/Mea

s

III –

Dat

a Ana

lysis/

Stats

IV -

Numbe

r Sen

se

V - Num

& A

lg Ope

r .

VI - P

rob/

Discre

te

Mathematics Standards

Pe

rce

nt % Above

% At

% Under

State BRange of Knowledge CorrespondenceHigh School Mathematics (N=51 items)

Standards# Hits # Objs Hit % Objs Hit Rng. of Know.

TitleGoals

#Objs

#Mean Mean Mean

I - Patterns, Relationships and Functions

2 11 10.44 4.22 38 NO

II - Geometry and Measurement

3 18 13 5.78 32 NO

III - Data Analysis and Statisti

3 14 13.44 5 35 NO

IV - Number Sense and Numeration

3 14 2.78 2.44 17 NO

V - Numerical and Algebraic Operations and Analytical ...

2 9 10.67 5.22 55 YES

VI - Probability and Discrete Mathematics

2 11 6.89 3.67 33 NO

Total 15 78.11 57.22 4.39 35

State B Range of Knowledge

0

20

40

60

80

100

I - P

atte

rns/

Fcns

II - G

eom

/Mea

s

III –

Dat

a Ana

lysis/

Stats

IV -

Numbe

r Sen

se

V - Num

& A

lg Ope

r .

VI - P

rob/

Discre

te

Mathematics Standards

Pe

rce

nt

State BBalance of RepresentationHigh School Language Arts (3 of 12 standards)(N=116 items)

Standards

Balance IndexBal. of

Represent. % Hits in Std/Ttl Hits

Index

TitleGoals

#Objs

#Mean S.D. Mean S.D.

I. - Meaning and Communication—Reading

1 5.11 28 8 0.57 0.12 NO

II. - Meaning and Communication—Writing

1 4 48 7 0.68 0.14 WEAK

VIII. - Genre and Craft of Language

1 5 17 6 0.63 0.16 WEAK

Total 12 55.33 8 18 0.36 0.21

State B Balance of Representation

0 50 100 150 200 250 300 350 400

I.

1.1

1.3

1.5

2

2.2

2.4

Lan

gu

age

Art

s S

tan

dar

ds

Number of Hits

What considerations should be given to different item types?

How to consider a multiple-point assessment item?

What is the trade off between multiple-choice items and open-ended items?

What are issues related to vertical alignment?

Appropriate progression of complexity across grades

Appropriate progression of content across grades

State A Mathematics DOK Levels for Objectives by Grade

0%

20%

40%

60%

80%

100%

3 4 5 6 7 8 10

Grade

Pe

rce

nt

of

DO

K L

ev

els

DOK Level 4

DOK Level 3

DOK Level 2

DOK Level 1

State A Reading Language Arts DOK Levels for Objectives by Grade

0%

20%

40%

60%

80%

100%

3 4 5 6 7 8 10

Grade

Per

cen

t o

f D

OK

Lev

els

DOK Level 4

DOK Level 3

DOK Level 2

DOK Level 1

Vertical Alignment Questions

What level of concurrence is there between objectives for the two grades?

To what extent do comparable objectives increase in depth from one grade to the next?

To what extent does the range of content increase from one grade to the next?

How does the balance of representation change from one grade to the next?

Type of Vertical Relationships

Broader Deeper Prerequisite New Identical