Integrated Document Analysis Workshop
description
Transcript of Integrated Document Analysis Workshop
Integrated Document Analysis Workshop
Peter Worth, Assessment Standards & Development Services, WestEd, CAJohn L. Smithson, Director, Measures of the Enacted Curriculum
Wisconsin Center for Education Research, University of Wisconsin-Madison
Washington, DC May 5-7, 2008
Describing the Content & Language Demands of
Standards, Assessments & Student Work
The intended curriculum: State content standards—What students should learn
The enacted curriculum: What teachers teach
The learned curriculum: Student outcomes based on school learning
The assessed curriculum: State (and other) assessments—tested learning
Analyzing Alignment
Linquistic Complexity by Language Demands
A neutral linguistic taxonomy
A neutral content language
Topics by Cognitive Demand
Content and Language Coding Form
Focus: (A = Academic/S = Social) Language Demand: (A-P, Z) Density: (L = Low/H= High)
Mode: Rec. = Receptive (LI=Listening; RD= Reading) Construction: (S=Simple/C=Complex)
Pro. = Productive (SP=Speaking, WR=Writing) Overall Complexity: (1= lowest, 4=highest;
X= Complexity cannot be determined)
Content Codes
Language Codes
Strand ID
TPC1
CGD1
Focus
(A/S)
Mode Language Demand
(A-P, Z)
Density
(L/H)
Construction
(S/C)
Overall Complexity
(1-4, X)
Comments
Rec. Pro.
1
2
3
4
5
6
Content and Language Coding
Grade 8 mathematics item
Content and Language Coding Form
Focus: (A = Academic/S = Social) Language Demand: (A-P, Z) Density: (L = Low/H= High)
Mode: Rec. = Receptive (LI=Listening; RD= Reading) Construction: (S=Simple/C=Complex)
Pro. = Productive (SP=Speaking, WR=Writing) Overall Complexity: (1= lowest, 4=highest;
X= Complexity cannot be determined)
Content Codes
Language Codes
Strand ID
TPC 1
CGD1
Focus
(A/S)
Mode Language Demand
(A-P, Z)
Density
(L/H)
Construction
(S/C)
Overall Complexity
(1-4, X)
Comments
Rec. Pro.
1 16 506 E A RD D L C 2
2 204 C E
3 K
4
5
6
K-12 Mathematics Content
K-12 Mathematics ContentBasic Algebra
The Goal
To render quantitative descriptions of instruction, standards, and assessments using a common language in order to facilitate comparisons and analyses of the three domains of a standards-based approach to education reform and their relationship to one another.
To Facilitate Comparisons
Academic Content
ELL Lang. Acquisition
Academic Content
ELL Lang. Acquisition
Academic Content
ELL Lang. Acquisition
The Two Dimensions Of Content
What students should know[Topics]
And…
Be Able to Do [Expectations for student performance]
Describing Instructional Content
SEC utilizes a two-dimensional taxonomy based on:Topic
by Cognitive Demand
1
2
3
4
5
B C D E F
Coarse Categories of Cognitive Demand
Content Areas
Memorize Perform Procedures
Demonstrate Understanding
Conjecture, Hypothesize
Non-routine problems
Number Sense
Operations
Measure-ment
Basic Algebra
Advanced Algebra
Geometric Concepts
The Content Matrix
Fine Categories of Cognitive Demand
Topics Memorize Perform
Procedures Demonstrate
Understanding Conjecture, Hypothesize
Non-routine problems
Place Value
Whole Nbrs.
Fractions
Decimals
Percents
Ratio & Proportion
The Content Matrix
… adding levels of relative emphasis yields a 3-D construct
Coarse Categories of Cognitive Demand
Content Areas
Memorize Perform Procedures
Demonstrate Understanding
Conjecture, Hypothesize
Non-routine problems
Number Sense
Operations
Measure-ment
Basic Algebra
Advanced Algebra
Geometric Concepts
Describing the Cognitive Domain
3
4
5
6
How Many Categories?
Bloom’s
SEC
DOK (Webb)
SCASS Science
Dimensions of Knowing & Inquiry
Acquire Use Extend
(From: Dimensions of Knowing and Inquiring about Science, State Collaborative on Assessments & Student StandardsScience Project, Council of Chief State School Officers, 1997)
Depth of Knowledge
* Webb, N. 1999. Alignment of Science and Mathematics Standards in Four State. NISE Research Monograph #18. Madison:Wisconsin Center for Education Research.
Skill/Concept
Recall
StrategicThinking
ExtendedThinking
Level
1
2
3
4
Skill/ConceptRecallStrategicThinking
ExtendedThinking
Acquire Use Extend
Exploring Cognitive Demand
Cognitive Demand (or Expectations for Student Performance)
Skill/ConceptRecallStrategicThinking
ExtendedThinking
Acquire Use Extend
MemorizePerform
ProceduresDemonstrate
Understanding
Conjecture,Generalize
Prove
Solve non-routine/ makeconnections
MemorizeConduct
InvestigationsCommunicateUnderstanding
AnalyzeInformation
Apply concepts/make
connections
RecallPerform
ProceduresAnalyze/
InvestigateEvaluate
Generate/Demonstrate
Cognitive Demand (or Expectations for Student Performance)
Skill/ConceptRecallStrategicThinking
ExtendedThinking
Acquire Use Extend
MemorizePerform
ProceduresDemonstrate
UnderstandingAnalyze
InformationEvaluate/Apply
Recall Understanding Application Analyze Evaluate Create
Practice Session #1
ExploringCognitive Demand
But first….
Let’s take a 10 minute break!
Exploring Cognitive Demand
CgD Immersion Activity• Organize into Groups/Tables• Each Table w/ CgD Pie• Each Person w/ Cgd Descriptors
Step 1: Place CgD cards on Pie Slices face-down
Step 2: Turn cards over: ID agreements e.g. 2 cards w/ same descriptor in same sliceif Group Agrees ... discuss key wordsif not … Discuss … operational definition to distinguish
Step 3: Discuss disagreementsif consensus reached put in envelope / if not, set aside
Content Analysis Materials
• Cognitive Demand List• Topics Lists• Comments & Suggestions Worksheet• Coding Forms• Documents to be analyzed
Cognitive Demand Lists
• Five categories of cognitive demand• Slightly different for each subject• Each category is defined by a list of descriptors• The list of descriptors are not exhaustive
• Each category stands on its own• Each category has an associated letter (B-F)
Dimension A: Content Topics
Topics List (In your packet of material)
Organized at two levels:Content Areas
(16 for Mathematics)(27 for Science)(14 for ELAR)
Topics (identified by number)(182 Mathematics Topics)(211 Science Topics)(114 ELAR Topics)
Plus: non-specific & other
Topics Lists
• Topics Lists• Mathematics• Science• English Language Arts & Reading
• Cover grades K-12• Organized into Content Areas• Topics & Content Areas have an associated #
Comments & Suggestion Worksheet
• One for each reviewer - more available• Use to:
• Record coding conventions/decision rules• Suggest/identify additional topics not listed• Suggest/identify additional CGD descriptors• Provide other comments & suggestions
• Be sure to turn in at end of workshop (and with mail-in materials, as necessary).
Coding Procedures
TPC CGDContent Descriptions arerecorded using pairs ofcodes to identifyparticular intersectionsTopic & CognitiveDemand Categories.
B200203
C
Cognitive Demand Categories arerepresented by letters B-F.
Topics are identifiedusing numbers.
D290
TPC CGD
Large Grain Topic Descriptions:To reference vague content descriptions thatcould cover most topics in a given contentarea, use the content area code (ends in 00)to indicate the relevant content area. (Thereare 7 content areas in K-8 Mathematics and25 content areas in K-8 Science.)
200Grain Size can berepresented using thefollowing conventions:
Coding Conventions
Should none of five categories of cognitivedemand listed on the Cognitive DemandCoding Sheet appear relevant, (e.g. thestandard statement makes no reference tocognitive demand, use the letter ‘Z’. Thisshould only occur rarely, if at all.
Non-SpecificCognitive Demand:
Coding Conventions
TPC CGD200 Z
Coding Conventions
Non-specific Content TPC CGD
0
Z
999 Z
Non-specific target subject 0
EOut of subject area
Cognitive Demand only
(*Please make note on your comments form.)
Non-specific Topics: TPC CGDB290
293 DOther: (rater-defined*)
Other: (topic not listed)
Coding Conventions
The Content Analysis Process
‘Coding’ Teams of 4-5 Content Experts
Independent Coding by each Analyst w/ Group Discussion
Goal for Process: Generalizability not Inter-rater Reliability
Should not be necessary to discuss every item – select by team
Pick-up and return documents / coding sheets to Alissa
Sign & return to Alissa non-disclosure forms
Practice Exercise #1 (Math)
Practice Exercise #2 (ELAR)
A neutral content language
Topics by Cognitive Demand
The intended curriculum: State content standards—What students should learn
The enacted curriculum: What teachers teach
The learned curriculum: Student outcomes based on school learning
The assessed curriculum: State (and other) assessments—tested learning
Content Analysis Workshop
Coding Forms• Assessment Coding Forms• Standards/Benchmark Coding Forms• Each is used to record content descriptions• Each content description consists of
• A topic number• A cognitive demand category letter
• The Integrated Analysis Coding Form• Combines Content & Linguistic Information
in a single data collection form
Rater: ________ Subject:__NC Math Gr 3 Benchmark Test (SF)_ Form: Test 1 Page of
Itm. Desig./Nbr. Itm. Desig./Nbr.TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
1 40
2 41
3 42
4 43
5 44
6 45
7 46
8 47
9 48
10 49
11 50
12 51
13 52
14 53
15 54
16 55
17 56
18 57
19 58
20 59
21 60
22 61
23 62
24 63
25 64
26 65
27 66
28 67
29 68
30 69
31 70
32 71
33 72
34 73
35 74
36 75
37 76
38 77
39 78
Content Code 2 Content Code 3Content Code 1 Content Code 2 Content Code 3 Content Code 1
Assessment Coding Forms
Rater: ________ Document:__NAEP Grade 8____Itm. Desig./Nbr. Itm. Desig./Nbr.
Number PropertiesNumber Sense TPC1 CGD1 TPC2 CGD2 TPC3 CGD31a 4a
1b 4b
1d 4c
1e 4d
1f
1g 5a
1i 5b
1j 5c
Estimation TPC1 CGD1 TPC2 CGD2 TPC3 CGD3 5d2a
5e2b
5f2c
Measurement
2d Measuring Physical Attributes TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
1bNumber Operations TPC1 CGD1 TPC2 CGD2 TPC3 CGD3
3a 1c
3d 1g
3e 1h
3f 1j
3g 1k
Properties of Number & Operations
TPC1 CGD3
CGD3CGD1 TPC2
CGD1 TPC2 CGD2 TPC3
CGD2 TPC3
Content Code 1 Content Code 2 Content Code 3 Content Code 1 Content Code 2 Content Code 3Ratios & Proportional Reasoning
TPC1
Standards Coding Forms
Standards Coding Forms
Nature of ScienceScience & Technology
Sci., Health, & Envrnmnt.Measurement in Science
Components of Living Sys.Biochemistry
BotanyAnimal Biology
Human BiologyGeneticsEvolution
Reproduction & Dev.Ecology
EnergyMotion & Force
ElectricityWavesKinetics & Equilibrium
Properties of MatterEarth Systems
AstronomyMeteorology
Elements & The Per. Sys.Chem. Form. & Reactions
Acids, Bases, & SaltsOrganic ChemistryNuclear Chemistry
Other:
All Content Areas
-0.018--0.008 -0.008-0.002 0.002-0.012 0.012-0.0220.022-0.032 0.032-0.042 0.042-0.052 0.052-0.0620.062-0.072 0.072-0.082 0.082-0.092
Mem
orize
Perform
Comm
unicate
Analyze
Apply
Nature of Science
Science & Technology
Sci., Health, & Envrnmnt.
Measurement in Science
Components of Living Sys.
Biochemistry
Botany
Animal Biology
Human Biology
Genetics
Evolution
Reproduction & Dev.
Ecology
Energy
Motion & Forces
Electricity
Waves
Kinetics & Equilibrium
Properties of Matter
Earth Systems
Astronomy
Meteorology
Elements & The Per. Sys.
Chem. Form. & Reactions
Acids, Bases, & Salts
Organic Chemistry
Nuclear Chemistry
Other:
All Content Areas
-0.018--0.008 -0.008-0.002 0.002-0.012 0.012-0.0220.022-0.032 0.032-0.042 0.042-0.052 0.052-0.0620.062-0.072 0.072-0.082 0.082-0.092
Gr. 5-8 National Science Standards