Creating Tests for Higher Order...

Creating Tests for Higher Order Thinking*

Thinking Schools Academy

*Content informed by:Gareis, C. R., & Grant, L. W. (2008). Teacher-made assessments.: How to connect curriculum, instruction, and student

learning. Larchmont, New York: Eye on Education

Th

ink

ing

Sch

oo

ls A

cad

emy

Lesson Objectives:

1. The learner will (TLW) understand that tests must assess lower- and higher-order thinking

(DOK1 - 10% emphasis)

2. TLW use a Table of Specifications to create lesson objectives, identify thinking level for each objective, and determine the lesson emphasis for the objective

(DOK 3 - 50% emphasis)

3. TLW understand guidelines for writing test questions (DOK 2 - 30% emphasis)

4. TLW will be introduced to the use of rubrics


Th

ink

ing

Sch

oo

ls A

cad

emy

Th

ink

ing

Sch

oo

ls A

cad

emy

Table of Specifications for this Lesson Thinking (Rigor) Level

Lesson Objectives: DOK1Recall

DOK2Application

DOK3Strategic Thinking

1. The learner will (TLW) understand that tests must assess lower- and higher-order thinking. 10% emphasis

High

2. TLW use a Table of Specifications to create lesson objectives, identify thinking level for each objective, and determine the lesson emphasis for the objective.

50% emphasis

High High

3. TLW understand guidelines for writing test questions.

30% emphasis

High Medium

4. TLW will be introduced to the use of rubrics.

10% emphasis

High

Lesson Objective #1

The learner will (TLW) understand that tests must assess lower- and higher-order thinking


Th

ink

ing

Sch

oo

ls A

cad

emy

4

The Planning Teaching & Learning Assessment Cycle

Planning

Establish What to Teach; Plan the Test; Create the Lessons

Teaching & Learning

Teach the Lessons while Checking for Learning

(Formative Assessment)

AssessmentAdminister the Test

(Summative Assessment)

Th

ink

ing

Sch

oo

ls A

cad

emy

Curriculum and Assessment Must Match

Curriculum Assessment

Th

ink

ing

Sch

oo

ls A

cad

emy

Test Student Thinking…

• To be a Thinking School we must test a wide range of thinking beyond the ability to memorize a correct answer.

• A good test assesses both lower- and higher-order thinking.

Th

ink

ing

Sch

oo

ls A

cad

emy

Test Both Lower- and Higher-Order Thinking

Th

ink

ing

Sch

oo

ls A

cad

emy

DOK3

Strategic

Thinking

DOK4

Create

DOK2

Application

DOK1

Recall LowerOrder

HigherOrder

Lesson Objective #2

TLW use a Table of Specifications to create lesson objectives, identify thinking level for each objective, and determine the lesson emphasis for the objective (DOK 3 - 50% emphasis)

(TLW means “The Learner Will”)

Th

ink

ing

Sch

oo

ls A

cad

emy

Planning Teaching and Assessment

• Before teaching, decide the following:

• What objectives will be taught and how much of the unit will each objective require (10%, 25%, 50%?)

• What will be the emphasis of each objective in the lesson? (high, medium, low)

• At what DOK level will the objective be taught? (DOK 1, 2, 3, 4)

• How will objectives be assessed? Th

ink

ing

Sch

oo

ls A

cad

emy

USING A TABLE OF SPECIFICATIONSA table of specifications is a tool for planning the lesson and the test by:

1. Creating Lesson objectives

2. Identifying thinking (DOK or rigor) levels for each objective

3. Planning the test

Th

ink

ing

Sch

oo

ls A

cad

emy

The Table of Specifications is a planning tool which links lesson objectives, rigor, and testing

Th

ink

ing

Sch

oo

ls A

cad

emy

Assessment

Rigor

Lesson Objectives

Table of Specifications

• A planning tool

• Addresses three questions:

• What is the content to be taught?

• At what thinking level will the content be taught?

• What will be the emphasis of the content?

Th

ink

ing

Sch

oo

ls A

cad

emy


Th

ink

ing

Sch

oo

ls A

cad

emy

Content

14


Th

ink

ing

Sch

oo

ls A

cad

emy

Content

Thinking Level by

DOK (Emphasis)

Recall

(DOK1)

Application

(DOK2)

Strategic

Thinking

(DOK3)

15


Th

ink

ing

Sch

oo

ls A

cad

emy

Content

Thinking Level by DOK and

Emphasis

Recall

(DOK1)

Application

(DOK2)

Strategic

Thinking

(DOK3)

1. Human archeological history (50% of test)

Describe hunter-gatherer societies

Identify locations

Identify tools

High

Emphasis

High

Emphasis

High

Emphasis

16


Th

ink

ing

Sch

oo

ls A

cad

emy

Content


Emphasis

Recall

(DOK1)

Application

(DOK2)

Strategic

Thinking

(DOK3)



Identify locations

Identify tools

High

Emphasis

High

Emphasis

High

Emphasis

2. Describe adaptations to climate changes

(25% of test)

Medium

Emphasis

Medium

Emphasis

Low

Emphasis

17


Th

ink

ing

Sch

oo

ls A

cad

emy

Content


Emphasis

Recall

(DOK1)

Application

(DOK2)

Strategic

Thinking

(DOK3)



Identify locations

Identify tools

High

Emphasis

High

Emphasis

High

Emphasis

2. Describe adaptations to climate changes

(25% of test)

Low

Emphasis

Medium

Emphasis

High

Emphasis

3. Describe eventual domestication of plants

and animals (25% of test)

Medium

Emphasis

Medium

Emphasis

Low

Emphasis

18

Activity:

• For this activity you will design your own Table of Specification for a lesson that you teach

Th

ink

ing

Sch

oo

ls A

cad

emy

19

Specifying your lessonDirections: Step 1 - for a lesson you teach, write 3 content objectives. Step 2: When thinking of the entire lesson, what percentage of lesson emphasis will each objective take? (all objectives should sum to 100%.)Step 3: for each DOK or rigor level, circle the emphasis for each objective.

Th

ink

ing

Sch

oo

ls A

cad

emy

20

Content

Objectives(The total of all content objectives should equal 100%)

Thinking Level by DOK

And Emphasis

Recall

(DOK1)

Application

(DOK2)

Strategic

Thinking

(DOK3)

Content Objective #1:

% of Lesson Emphasis: ______

(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High


% of Lesson Emphasis: _______

(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High


Th

ink

ing

Sch

oo

ls A

cad

emy

21

Content



and Emphasis

Recall

(DOK1)

Application

(DOK2)

Strategic

Thinking

(DOK3)



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

1. Begin by breaking your lesson down into

objectives. Write up to 3 objectives for your lesson


Th

ink

ing

Sch

oo

ls A

cad

emy

22

Content



and Emphasis

Recall

(DOK1)

Application

(DOK2)

Strategic

Thinking

(DOK3)



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

2. What percentage of the total lesson will be emphasized

with this objective?




Th

ink

ing

Sch

oo

ls A

cad

emy

23

Content



and Emphasis

Recall

(DOK1)

Application

(DOK2)

Strategic

Thinking

(DOK3)



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High



(Out of 100%)

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

Circle one:

Low/Medium/High

2. What percentage of the total lesson will be emphasized

with this objective?



3. For each objective, indicate

the emphasis as low, medium, or high by

DOK

Lesson Objective #3

• TLW understand guidelines for writing test questions (DOK 2 - 30% emphasis)

Th

ink

ing

Sch

oo

ls A

cad

emy

24

Making a Test: Matching DOK to type of Test Question

Th

ink

ing

Sch

oo

ls A

cad

emy

Question Types

Rigor Level

Multiple

Choice

True

False

Fill in the

Blank Matching

Short

Answer Essay

DOK1 Recall

DOK2 Application

DOK3 Strategic

Thinking

Writing Test Items

Th

ink

ing

Sch

oo

ls A

cad

emy

True/False Rules: (typically DOK1 questions)

#1. Only one idea should be in a True/False statement.

#2. Make the statement absolutely true or absolutely false.

#3. Avoid absolute qualifiers like always and never.

#4. Avoid using negatives in a True-False statement.

#5. Always avoid the use of opinion in statement items.

Matching Rules: (typically DOK1 questions)

#1. Use homogeneous content in a matching set.

#2. Place item to be matched on the right with the longer

responses on the left.

#3: Keep the matching set short.

#4: Use an uneven number of items to match or allow responses to be

used more than once.

#5. Order items in a logical manner.

Writing Test Items - Example

Th

ink

ing

Sch

oo

ls A

cad

emy

Which depth-of-knowledge level is application?

A. DOK1 (distractor item)

B. DOK2 (correct response)

C. DOK3 (distractor item)

D. DOK4 (distractor item)

Writing Multiple Choice Questions(can be DOK1, 2 or 3, but not 4)

Th

ink

ing

Sch

oo

ls A

cad

emy

Question Stem Rules

#1. Be sure the problem is written clearly.

#2. Do not give away the answer in the question stem.

#3. Write the stem from a positive perspective.

#4. Qualifiers such as most likely, best, least, and most should be

emphasized.

Rules for Writing Choices

#5. Write answers that are parallel in grammar and length.

#6. The answer choices should be plausible

#7: Try to avoid using “None of the above” and “All of the above.”

#8. Put your answers in a logical order.

#9. Do not put clues to the answer in your response choices.

#10. Be certain that there is only one correct response.

Writing Higher Order Questions

• A goal of the Thinking Schools Academy is to teach and assess higher order thinking

• Higher order (DOK3) thinking can be assessed using multiple choice, fill-in-the-blank, and short-answer questions.

Th

ink

ing

Sch

oo

ls A

cad

emy

5 Principles for Writing DOK3 Multiple Choice Items

1. Match the content with a rigor or thinking level (DOK 1, 2, 3, 4)

2. Use novelty

3. Focus on content that is complex

4. Use an extended prompt

5. Provide stimulus material

Th

ink

ing

Sch

oo

ls A

cad

emy

Principle #1: Match the Content with Cognitive Level

Th

ink

ing

Sch

oo

ls A

cad

emy

Learning Objective: Analyze the factors influencing where hunter-gatherer societies locate. (DOK3 learning objective)

Poor Question Item:Which factor below would NOT be used to determine where a hunter-gatherer community might locate?

A. Availability of waterB. Proliferation of edible plantsC. Amount of retail shoppingD. Acceptable climate

Better Question:A group of hunter-gatherer leaders were considering whether to relocate their community. One leader pointed out that the wild animals they hunted for food had significantly decreased in number while another suggested that a different location with better natural beauty would make life more enjoyable.

Which of the following most likely had the greatest influence on their decision to relocate the community?

A. Better natural beautyB. Cleaner airC. A decrease in wild animalsD. A safer community

Principle #2: Introduce Novelty

• To test DOK1 knowledge, use straight forward questions that will be familiar to students.

• To assess higher order thinking, use novelty that will be less familiar to students.

• The previous slide is an example where a situation was created (the meeting of hunter-gatherer leaders) to introduce novelty

• The novelty requires students to analyze or evaluate the situation based on their previous learning

Th

ink

ing

Sch

oo

ls A

cad

emy

Principle #3: Focus on Complex Content

• Simple content focuses on one dimension or just a few components.

• A list of dates is simple material even though they may be difficult to memorize.

• An explanation of events involving the dates could be complex.

• Simple content is assessed at DOK1 & 2 levels

• Complex content is assessed at DOK3 levels

Th

ink

ing

Sch

oo

ls A

cad

emy

Example of a DOK1 and DOK3 Multiple Choice Question for the Same Lesson Objective

Th

ink

ing

Sch

oo

ls A

cad

emy

Lesson Objective: The student will understand the variables that contribute to a sustainable hunter-gatherer community.

DOK1 Question:Which of the following IS NOT a variable contributing to a successful hunter-gatherer community?A. WaterB. Edible plantsC. Hospitable climateD. Well-paved roads

DOK3 Question:The typical spring and summer has resulted in much less rainfall than normal. This will most likely result in:A. A hot summerB. Fewer healthy plants and

vegetables to harvestC. A large harvest of edible plants

and fruitD. Trees which have grown faster

than normal

Activity:

• Discuss with your “elbow partner” what distinguishes DOK1 from DOK2 from DOK3 test questions?

• Make a bullet list of your conclusions

Th

ink

ing

Sch

oo

ls A

cad

emy

Lesson Objective #4

• TLW will be introduced to the use of rubrics


Th

ink

ing

Sch

oo

ls A

cad

emy

Developing a Rubric to Assess Complex Tasks

• A rubric lists the scoring criteria for a task

• Rubrics are used to decrease subjectivity in grading a complex task

• Articulating essential criteria helps the teacher to make grading more objective

Th

ink

ing

Sch

oo

ls A

cad

emy

An Example of a Rubric to Grade Bicycle Riding

Th

ink

ing

Sch

oo

ls A

cad

emy

Indicator 1 2 3 4 Score

Balance Immediately falls off bike

Can maintainbalance for up

to 5 meters

Generallymaintains

balance

Balance is rarely lost

Steering No steering ability

Steers in a wobbly manner over very short

distances

Maintains good control in most

riding circumstances

Steering is lost only under

extreme circumstances

Pedaling Is not able to pedal

Can pedal softly for up to 5

meters

Can pedal with little thought; can

easily adjust pedaling speed

Expert at pedaling in a

variety of conditions

Control Cannot maintain control of

bicycle

Control is very limited

Can generally maintain control

of bicycle

Maintainscontrol in all

terrain conditions

Total Score:(Add the scores for balance, steering, pedaling, and control)

Applications for Rubrics

• Scoring a writing assignment

• Grading a performance

• Scoring a research report

• Evaluating a speech

• Evaluating the creation of something new such as a piece of art, a product improvement or innovation T

hin

kin

g Sc

ho

ols

Aca

dem

y

Activity: Design Your Own Rubric

Th

ink

ing

Sch

oo

ls A

cad

emy

Indicator 1 2 3 4 Score

Total Score:(Add the scores for each row)

Summary

• Assessments must match the curriculum

• Curriculum must include Higher Order Thinking (DOK3)

• Decide the percentage of curriculum dedicated to DOK1, DOK2, and DOK3

• Decide the emphasis for each curriculum objective (low, medium, high)

• Use the Table of Specifications as a planning tool

Th

ink

ing

Sch

oo

ls A

cad

emy

Creating Tests for Higher Order...

Documents

Transcript of Creating Tests for Higher Order...