1

Bias and Sensitivity Review of Items for the

MSP/HSPE/EOC

August, 2012

ETS Olympia1

General WCAP Development Cycle

Operational Scoring Operational Rangefinding Operational Testing

Data ReviewPilot Scoring Training and Pilot

ScoringPilot Rangefinding

Item Pilot Testing Item Reviews for Content & Bias/Sensitivity by Committee

Item Reviews for Content by OSPI

Item Writing Training and Item Writing

Develop Item and Test Specifications

Principles of Universal Design

Assessments should be designed to be valid and accessible for use by the widest possible range of students, including students with disabilities and students with limited English proficiency.

Results should not be influenced by disability, ethnicity, gender, or English language ability.

3

What is Item Bias?

Language or content in the item that prevents members of a group from demonstrating their knowledge and skills in a content area; and

Systematic error in the measurement process.

4

Potential Sources of Bias and Stereotyping

Regional and geographic differences in language usage and topic familiarity

Linguistic issues (e.g., idiomatic expressions or figurative language)

Gender and age stereotypes Ethnic, cultural, and religious stereotypes Socio-economic and occupational

stereotypes5

Sensitivity IssuesAny reference or language in an item or passage that might cause a student to have an emotional reaction during the test administration and prevent the student from being able to accurately demonstrate knowledge and skills.

Examples found in previous bias & sensitivity reviews: Passage about abuse or struggles of an individual or group because

of minority status Change of meaning or structure in a culture’s legend or folk tale Competitive context in passages or items Controversial issue unrelated to the content standards Use of patronizing language to describe an individual or group

6

How Might Potential Item Bias be Detected?

Judgmental Procedure: Review of item for issues related to context, language, or stereotyping prior to field-testing

Statistical Procedure: DIF (Differential Item Functioning) analysis following field-tests using item statistics

7

Purpose of Bias and Sensitivity Review

Review test items for potential sources of bias, sensitive issues, and gender or cultural stereotypes.

Apply professional test development standards to ensure items are fair and not insensitive or offensive.

8

Read the Bias & Sensitivity Review Guidelines

Questions?

Comments?

Guiding Questions

Is there anything controversial, inflammatory, or insensitive in a scenario, reading passage, or item context?

Is the language likely to be familiar to most examinees regardless of socioeconomic status, region, language proficiency, cultural background, or gender

Is there any apparent patronizing language, stereotypical descriptions, or stereotypical actions/beliefs present?

Would an item context or content give students of a particular group, background, or region a distinct advantage or disadvantage?

10

Sample Item #1In the poem “The Bells,” what is the effect of phrases such as the jingling and the tinkling and the moaning and the groaning?

A. They indicate the actual sounds made by the bells.

B. They reflect the beauty of the sounds made by the bells.

C. They provide the required number of syllables in each line.

D. They show that the speaker is gradually losing touch with reality.

Bias issue: not accessible to students who are deaf or hard of hearing

11

Sample Item #2

Which of the following measures could be the length of a typical Par 4 golf fairway? A. 400 inches B. 400 feet C. 400 yards D. 400 miles

Bias issue: SES and/or prior knowledge

12

Sample Item #3 At a carnival, Rolando sees a booth that has a prize

wheel. The wheel has 20 sections, and 5 of these sections are labeled ‘Winner.’ If Rolando decides to give the wheel booth a shot, what is the probability that he will win a prize?

A. 0.20

B. 0.25

C. 0.33

D. 0.75

Bias issue: Idiomatic Language “… give the wheel booth a shot…”

13

Practice Item Review Read the items Decide whether the content of the items

have any of these problems: Stereotyping or negative representation of any

group

Context that is controversial or sensitive

Language or context that may be unfamiliar to

a student because of region, language

proficiency, socio-economic status, etc.

Do you need to review for: Names of the children and adults in math

and science items? NO, names are adjusted for gender and ethnic balance once tests are put together

Whether the concepts or skills are appropriate for students at the identified grade level?NO, content review committees check that items only measure on-grade level concepts and skills

Read the passages, keeping in mind the guiding questions. Make your own judgment. Write your judgment on the

review form: Accept as is (no apparent bias/sensitivity issue) Edit Reject due to potential bias or sensitivity issue

As a group, discuss each item flagged for revision or rejection.

Recorder will take detailed notes. Make a group decision about each item. Record dissenting position, if appropriate

How To Proceed

16