Tools for Supporting Increased Rigor & High-Quality Assessment

and Instruction

Verona Public Schools

Karin K. Hess, Ed.D.www.karin-hess.com

Track your reflections as we work…

Ways I am refining my thinking about DOK/rigor…•?•?

Scaffolding strategies for getting students to deeper thinking…•?•?

Before we begin…

• Take a minute to jot down words/phrases that come to mind when you think of “cognitive rigor” as it relates to instruction, learning, and/or assessment.

Let’s apply your rigor definitions

Your class has just read some version of Little Red Riding Hood.

• What is a basic comprehension question you might ask?

• What is a more rigorous question you might ask?

The Hess Cognitive Rigor Matrix integrates Bloom + Webb

Different states/schools/teachers use different models to describe cognitive rigor. Each addresses something different.

• Bloom – What type of thinking (verbs) is needed to complete a task?

• Webb – How deeply do you have to understand the content to successfully interact with it? How complex is the content?

Merging Bloom + Webb:

The thinking behind the development of the Hess Cognitive Rigor Matrix …

Bloom’s Taxonomy [1956 ] & Bloom’s Cognitive Process Dimensions [2001]

Knowledge -- Define, duplicate, label, list, name, order, recognize, relate, recall

Remember Retrieve knowledge from long-term memory, recognize, recall, locate, identify

Comprehension -- Classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, review, select, translate

Understand -- Construct meaning, clarify, paraphrase, represent, translate, illustrate, give examples, classify, categorize, summarize, generalize, predict…

Application -- Apply, choose, demonstrate, dramatize, employ, illustrate, interpret, practice, write

Apply -- Carry out or use a procedure in a given situation; carry out or use

/apply to an unfamiliar task Analysis -- Analyze, appraise, explain calculate, categorize, compare, criticize, discriminate, examine

Analyze -- Break into constituent

parts, determine how parts relate Synthesis -- Rearrange, assemble, collect, compose, create, design, develop, formulate, manage, write

Evaluate -- Make judgments based on criteria, check, detect inconsistencies/fallacies, critique

Evaluation -- Appraise, argue, assess, choose, compare, defend, estimate, explain, judge, predict, rate, core, select, support, value

Create -- Put elements together to form a coherent whole, reorganize elements into new patterns/ structures

Webb’s Depth-of-Knowledge Levels• DOK-1 – Recall & Reproduction - Recall of a fact, term, principle,

concept, or perform a routine procedure

• DOK-2 - Basic Application of Skills/Concepts - Use of information, conceptual knowledge, select appropriate procedures for a task, two or more steps with decision points along the way, routine problems applying 2+ concepts, organize/display data, interpret/use simple graphs

• DOK-3 - Strategic Thinking - Requires reasoning, developing a plan or sequence of steps to approach problem; requires some decision making and justification; abstract, complex, or non-routine; often more than one possible answer or approach

• DOK-4 - Extended Thinking - An original investigation or application to real world; requires time to research, problem solve, and process multiple conditions of the problem or task; non-routine manipulations, across disciplines/content areas/multiple sources

DOK Misconception #1:

All kids can’t do this; or Kids don’t need scaffolding to get “up” there.

Engaging in “a complex task” with supports/ scaffolding is an essential step along the way to proficiency (Vygotsky’s ZPD)

– Do it with others first; DOK 3 and 4 are not meant to only be done alone/independently, especially at first

– Oral language & meaningful discourse support deeper thinking and increase initial exposures to the content and student engagement. This is NOT cheating!

– One strategy: Plan questioning & formative probes from DOK 1-2-3-4 over the course of a lesson or unit of study. Consider all DOK levels in your planning.

Karin Hess (2008). Using learning progressions as a schema to monitor progress across grades.

Vygotsky: Zone of Proximal DevelopmentVygotsky: Zone of Proximal Development(What a child can do (What a child can do with assistancewith assistance today) today)

LEARNING PROGRESSIONS ZONE: Dynamic area

Causes development to move forwardSocial interaction essential (scaffolding)

Actual Development

Area

Potential Development

Area

The ZONE

What a child can do independently now: “ENTRY”

What a child can do independently tomorrow/future

DOK Misconception #2:

Webb’s DOK model is a taxonomy• Bloom’s is a taxonomy, intended to be a

hierarchy

• Primary Weaknesses of Bloom: generic verbs (void of content); same verbs at different levels

• Webb’s DOK model is nominative: – It names how you interact with content

– It differentiates varying levels of engagement with content and suggests what tasks might look like

– DOK 4 is not better than DOK 3, or DOK 2, or DOK 1

DOK Misconception #3:

Bloom verbs & levels = Webb DOKThe DOK “Wheel of Misfortune” implies that a DOK level is

indicated by a particular verb or set of verbs.

Norman Webb, “It’s what comes after the verb, that indicates the complexity of a task.”

DOK Misconception #4:

DOK is about difficulty.

• The intended student learning outcome determines the DOK level. What mental processing must occur? DOK = Complexity, not difficulty!

• While verbs may appear to point to a DOK level, it is what comes after the verb that is the best indicator of the rigor/DOK level and complexity of the task.– Describe the information contained in graphics or data tables in the text; or the

rule for rounding a number– Describe how the two story characters are alike and different.– Describe the data or text evidence that supports your solution, reasoning, or

conclusions– Describe varying perspectives on global climate change using supporting

scientific evidence, and identify the most significant effects it might have on the planet in 100 yrs.

“UGs”

Evaluate

Recall, locate basic facts, definitions, details and eventsSelect appropriate word when intended meaning is clear

DOK LEVEL 1Recall and Reproduction

Understand

DOK LEVEL 3Reasoning

DOK 2Skills and Concepts

DOK 4Extended Thinking

Create

Use context to find meaningObtain and use information in text features

Explain relationshipsSummarizeCentral ideas

AnalyzeApply

Use language structure or word relationships (synonyms/anto-nyms)

Remember

Analyze or interpret author’s craft (e.g., literary devices, viewpoint, or potential bias) to critique a text

Use concepts to solve non-routine problems and justify solutions with evidence

Explain, generalize or connect ideas using supporting evidence (quote, text, evidence, data, etc.)

Compare literary elements, facts, terms and events.Analyze format, organization & text structures

Identify information in a graphic, table, visual, etc.

Cite evidence and develop a logical argument for conjectures based on one text or problem.

Synthesize information across multiple sources Articulate a new voice, theme, perspective.

Evaluate relevancy, accuracy and completeness of information

Analyze multiple sources or textsAnalyze complex abstract themes.

Devise an approach among alternatives to research a novel problem

Explain how concepts relate to other content domains

Develop a complex model or approach for a given situationDevelop an alternative solution

Generate conjectures or hypotheses based on observations or prior knowledge

Brainstorm ideas, concepts, problems, or perspectives related to a topic

BloomBloom’’s Taxonomy s Taxonomy ++ Webb Webb’’s DOK = the Hess CRMs DOK = the Hess CRM 14

The Hess Cognitive Rigor Matrix Applies Webb’s DOK to Bloom’s Cognitive Process

DimensionsDepth + Thinking

Level 1Recall & Reproduction

Level 2Skills & Concepts

Level 3Strategic Thinking

Level 4Extended Thinking

Remember - Recall, locate basic facts, details, events

Understand - Select appropriate words to use when intended meaning is clearly evident

- Specify, explain relationships- summarize– identify main ideas

- Explain, generalize, or connect ideas using supporting evidence (quote, example, data …)

- Explain how concepts or ideas specifically relate to other content domains or concepts

Apply - Use language structure (pre/suffix) or word relationships (synonym/antonym) to determine meaning

– Use context to identify meaning of word- Obtain and interpret information using text features

- Use concepts to solve non-routine problems

- Devise an approach among many alternatives to research a novel problem

Analyze - Identify whether information is contained in a graph, table, text feature, etc.

– Compare literary elements, terms, facts, events– analyze format, organization, & text structures

- Analyze or interpret author’s craft (literary devices, viewpoint, or potential bias) to critique a text

– Analyze multiple sources- Analyze complex/abstract themes

Evaluate “UG” = unsubstantiated generalization

– Cite evidence and develop a logical argument for conjectures

- Evaluate relevancy, accuracy, & completeness of information

Create - Brainstorm ideas about a topic

- Generate conjectures based on observations or prior knowledge

- Synthesize information within one source, data set, or text

- Synthesize information across multiple sources or texts

DOK Misconception #5:

All DOK levels can be assessed with a multiple choice question

• That’s just dumb!• Weak DOK 3 multiple choice items are possible; but

does selecting the best option (e.g., locate supporting evidence for a theme) provide as much insight as seeing HOW a student formulates and reveals thinking?

• By their nature, DOK 3 and 4 questions/tasks are more open-ended, generally take longer to respond to/solve, and may have more than one “appropriate right answer”

2. The DOK Matrix Instructional Paths

Each standard has an assigned Depth of Knowledge.

The DOK determines the cognitive level of instruction.

Recall, locate basic facts, definitions, details, events

Select appropriate words for use when intended meaning is clearly evident.

DOK 1Recall and Reproduction

Remember

Understand

DOK 2Skills and Concepts

Apply

Explain relationshipsSummarizeState central idea

Use context for word meaningsUse information using text features

DOK 3Reasoning and

Thinking

Analyze

Analyze or interpret author’s craft (e.g., literary devices, viewpoint, or potential bias) to critique a text

Explain, generalize or connect ideas using supporting evidence (quote, evidence, data)

.

Cite evidence and develop a logical argument for conjectures based on one text or problem

Evaluate

Use concepts to solve non-routine problems and justify

DOK 4Extended Thinking

Synthesize across multiple sources/ textsArticulate a new voice, theme, or perspective

Evaluate relevancy, accuracy and completeness of information across texts or sources

Analyze multiple sources or multiple textAnalyze complex abstract themes

Devise an approach among many alternatives to research a novel problem

-Explain how concepts or ideas specifically relate to other content domains.

Develop a complex model or approach for a given situationDevelop an alternative solution

.Create

Instruction & Assessment Decisions…

17

Selected ResponseSelected ResponseSelected ResponseSelected Response

Constructed ResponseConstructed ResponseConstructed ResponseConstructed Response

Performance Performance TasksTasks

Performance Performance TasksTasks

Let’s practice using the CRM-back to Little Red Riding Hood

• Your sample questions – a basic and more rigorous question

• Handout #2: Linking Research, Tool #1: CRM template for Close Reading (page 4)

Depth + Thinking

Level 1Recall & Reproduction

Level 2Skills & Concepts

Level 3Strategic Thinking/ Reasoning

Level 4Extended Thinking

Remember What color was Red’s cape?Who is this story about?

Understand Who are the main characters?What was the story’s setting?

Retell or summarize the story in your own words.

What is the author’s message or theme? Justify your interpretation using text evidence.

Apply Identify words/phrases that helped you to know the sequence of events in the story.

Analyze Is this a realistic or fantasy story?

Compare the wolf character to the character of Red. How are they alike-different?

Is this a realistic or fantasy story? Justify your interpretation using text evidence.

Are all wolves (in literature) like the wolf in this story? Support your response using evidence from this and other texts.

Evaluate What is your opinion about the cleverness of the wolf? Justify your opinion using text evidence.

Which version has the most satisfying ending? (establish criteria first, then locate evidence)

Create Write text messages between Red & her mother explaining the wolf incident.

Some other content examples…

• Your class will be learning about…– Fractions/decimals; Data use– Data use; scientific investigation– Elements of art & principles of design

• Come up with a basic understanding and more rigorous question you could pose.

• Use a CRM Template to Plan Units

Depth + Thinking

Level 1Recall & Reproduction

Level 2Skills & Concepts

Level 3Strategic Thinking/ Reasoning

Level 4Extended Thinking

Remember What is slope?What is white space?

Understand Read, write, and represent these fractionsDescribe why white space is used.

Explain how you solved this problem.Why control variables in the investigation?

Construct an argument to show equivalence using area, set, and linear models

Apply Convert this fraction to a decimalAdd these fractions

Use these data to graph your solution

Conduct the investigation, interpret results, and support conclusions with data

Analyze What kind of graph or model is this?Which data point shows ____?Find examples…

Compare these methods.

Which graph shows how the data would be displayed?

Interpret what was happening in the event? Justify your interpretation using what you know about slope.

Analyze more than one product(same time period, medium,theme drawing from multiple contexts source materials for the analyses

Evaluate UG - Which team is the best?

How would you rank these ___? Justify your rankings using data that supports your criteria.

Some say the NFL settlement for player brain injury is not adequate. Evaluate both sides using data to determine the validity of this claim.

Create How would you demonstrate each technique?

Create a card game using fractions.Create scenario explained by a data display.

Integrate multiple source materials with intent to develop a product

DOK Misconception #6:

Higher order thinking = deeper learning

• What we have thought of as “higher order” (analysis, evaluation, creative thinking) might only be engaging or fun…and not always deeper

• Many critical thinking examples do not go deep or get to DOK 3 or 4 (e.g., interpret/solve and justify)

• Shift our thinking from “higher order” to deeper learning, and that can mean:– deeper understanding

– deeper application

– deeper analysis, etc.The Hess CRM illustrates this shift

Some general rules of thumb…• If there is one correct answer, it is probably level

DOK 1 or DOK 2– DOK 1: you either know it (can recall it, locate it, do it) or you don’t know it – DOK 2 (conceptual): apply one concept, then make a decision before going

on applying a second concept; express relationship (if-then; cause-effect)

• If more than one answer/approach, requiring evidence, it is DOK 3 or 4– DOK 3: Must interpret, provide supporting evidence and reasoning (not just

HOW solved, but WHY it works– explain reasoning for each step/decision made)

– DOK 4: all of “3” + use of multiple sources/data/ texts; initiate & complete an investigation

DOK Misconception #7:

Multi-step or longer tasks, multiple texts, or complex texts always means deeper thinking

• DOK 2 is not simply more than one step, it’s applying more than one concept; DOK 2 is still routine/typical (main idea, word problems, etc.)

• Simply reading more complex texts, but NOT delving deeply into the text’s meaning/style/etc., is likely to still be DOK 1 or 2

• DOK 3 requires some aspect of open-endedness and interpretation with justification or support; DOK 4 = drawing from multiple sources

Reading StandardsDepth + Thinking

Level 1Recall & Reproduction

Level 2Skills & Concepts

Level 3Strategic Thinking

Level 4Extended Thinking

Remember KEY DETAILSDecode, read orally

Understand KEY DETAILSWORD MEANINGS- fill in

CENTRAL IDEASSUMMARIZEpredict, infer

REASONING & SUPPORT – interpret theme, purpose, pt of view/perspective

REASONING & SUPPORT –multiple texts

Apply WORD STRUCTURE -roots, affixes, structure, RELATIONSHIPSsynonyms-antonyms

WORD MEANINGS-use in contextUSE TEXT STRUCTURES & FEATURES

TEXT STRUCTURES & FEATURES – how is the message structured/ presented?

REASONING & SUPPORT –multiple texts-compare treatments

Analyze USE of TEXT STRUCTURE or FEATURESLANGUAGE USE-identify non- literal usage

ANALYSIS & REASONING WITHIN TEXTSLANGUAGE USE-interpret non- literal usage

ANALYSIS & REASONING ACROSS TEXTS

Evaluate AUTHOR’s CRAFT WITHIN A TEXT (e.g., language use impact/intent; bias, reasoning)

EVALUATE AUTHOR’s PURPOSE or CRAFT ACROSS TEXTS

Create

Reading & Writing Depth + Thinking

Level 1Recall & Reproduction

Level 2Skills & Concepts

Level 3Strategic Thinking

Level 4Extended Thinking

Remember KEY DETAILS

Understand KEY DETAILSWORD MEANINGS- fill in

CENTRAL IDEASSUMMARIZEpredict, infer

REASONING & SUPPORT – DEVELOP theme, or pt of view/ perspective topic

REASONING & SUPPORT – use multiple texts - compare or elaborate

Apply WORD STRUCTURERELATIONSHIPS

EDIT/CLARIFYUSE TECHNOLOGYCITE SOURCES

WORD MEANINGS-LANGUAGE USEUSE OF TEXT STRUCTURES & FEATURES

INTEGRATE TEXT STRUCTURES & FEATURES into Compositions

REASONING & SUPPORT

Analyze USE of TEXT STRUCTURES or FEATURESLANGUAGE USE

ANALYSIS & REASONING WITHIN TEXTS – RESEARCH for WritingDevelop reasoning

ANALYSIS & REASONING ACROSS TEXTSRESEARCH for Writing;Comparing themes

Evaluate AUTHOR’s CRAFT WITHIN A TEXTEvaluate credibility of sources

EVALUATE AUTHOR’s PURPOSE or CRAFT ACROSS TEXTS

Create WRITE/EDIT BRIEF TEXTS

COMPOSE /REVISE FULL TEXTS

COMPOSE FULL TEXTS-sources

Math Content Standards & Math Practices

Depth + Thinking

Level 1Recall & Reproduction

Level 2Skills & Concepts

Level 3Strategic Thinking (support with data, equations, models, etc.)

Level 4Extended Thinking (cross domains)

Remember Know math facts, terms

Understand Attend to precisionEvaluate expressions, plot point

Model with mathematicsEstimate, predict, observe, explain relationships

Construct viable argumentsGeometry proof

Integrate concepts across domains

Apply Calculate, measure, make conversions

Make sense of routine problems

Make sense of non-routine problems

Design & conduct a project

Analyze Identify a patternLocate information in table

Use tools strategicallyClassify, organize data, extend a pattern

Reason abstractlyGeneralize a pattern

Analyze multiple sources ofevidence

Evaluate Critique the reasoning of others

Create Design a complex model

For each assessment task (or rubric)…ask

– What is its purpose? (What content/skill is being assessed? is there a ‘right’ answer?)

– What is the implied/intended rigor? (What mental processing would you expect students to engage in? Use the CRM to find descriptors.)

– Which standards does it REALLY assess? (content + intended rigor)

– Does the scoring guide/rubric match content + intended rigor?

– What would student responses tell a teacher if students could/could not do all or part of this task? (open-ended tasks, reasoning used) – next instructional decisions are clear

Take-Away Messages: Cognitive Rigor & Some Implications for Assessment

• Begin with daily DOK3 classroom discourse!• Assessing only at the highest DOK level (the

“ceiling”) will miss opportunities to know what students do & don’t know – go for a range; end “high” in selected/prioritized content

• Performance assessments can offer varying levels of DOK embedded in a larger, more complex task

• Planned formative assessment strategies and tools can/should focus on differing DOK levels

Some resources worth exploring…• www.nms.org – national math science initiative – gr 3 – HS; also has ELA and

SS performance tasks with DOK designations

• www.readworks.org – gr k-8; short literary & informational texts with CC Qs

• http://www.louisianabelieves.com/resources/library/common-core-state-standards - LA gr 3-HS math & ELA tasks (some math samples are weak and not all math DOK levels are correct, but generally good examples)

• http://ccgpsmathematicsk-5.wikispaces.com/K-5+Formative+Assessment+Lessons+%28FALs%29 - GA gr K-5 math

• www.exemplars.com – sample math PAs for k-12; science PAs for k-8

• Dan Meyer blog – math PAs for MS-HS; kids have to build the problems by deciding what’s needed to solve them – good strategic thinking required

• Hess & Gong (2014). Ready for college and career? Achieving the Common Core Standards and beyond through deeper, student-centered learning http://www.nmefoundation.org/resources/scl-2/ready-for-college-and-career

• Links on www.karin-hess.com: – Karin’s YouTube video – “Text Complexity Tools” (qualitative text analysis)

– Karin’s vimeo DOK video – http://vimeo.com/18281415

– Karin’s YouTube DOK video – excerpt from a CT workshop