Reflective Assessment/AlignmentReflective Assessment/Alignment

Daniel. T. HickeyDaniel. T. Hickey

Learning & Performance Support LabLearning & Performance Support Lab

Department of Educational PsychologyDepartment of Educational Psychology

University of GeorgiaUniversity of Georgia

AAAS/CCMS Knowledge Sharing InstituteAAAS/CCMS Knowledge Sharing Institute

June 2003June 2003

Two Questions about AssessmentTwo Questions about Assessment

Aligning assessments to standards:Aligning assessments to standards:– How does articulation of outcomes impact How does articulation of outcomes impact

assessment?assessment?– What is the impact of learning performances?What is the impact of learning performances?– Depends on the goal of assessment.Depends on the goal of assessment.

Impact of Assessment Practice:Impact of Assessment Practice:– How can information be used to revise How can information be used to revise

curriculum?curriculum?– What is the diagnostic power of items?What is the diagnostic power of items?– How can assessment advance learning?How can assessment advance learning?– Depends on the format of the assessment.Depends on the format of the assessment.

CCMS Design Tensions and CCMS Design Tensions and Curricular SolutionsCurricular Solutions

Learning inquiry vs. content coverage:Learning inquiry vs. content coverage:– Focus on coherent “slices” though the systems.Focus on coherent “slices” though the systems.– Tsou’s analogical comparisons.Tsou’s analogical comparisons.

Self-directed, intrinsically-direct inquiry Self-directed, intrinsically-direct inquiry leads to multiple paths:leads to multiple paths:– Focus on select benchmarks.Focus on select benchmarks.

Some problem contexts require non-Some problem contexts require non-targeted background knowledge:targeted background knowledge:– Make tradeoffs in context selection.Make tradeoffs in context selection.

The New Challenge of The New Challenge of “Accountability”“Accountability”

NCLB, WWC, likely DOE Technology Plan use NCLB, WWC, likely DOE Technology Plan use standardized M-C tests as policy lever:standardized M-C tests as policy lever:– Bizarre way to demonstrate knowledge.Bizarre way to demonstrate knowledge.– Cheap, reliable, & naïve appeal.Cheap, reliable, & naïve appeal.

Conventional curriculum & integrated teaching Conventional curriculum & integrated teaching systems have a profound advantage:systems have a profound advantage:– Fine-tuned to directly present many of the associations Fine-tuned to directly present many of the associations

needed to identify most correct response.needed to identify most correct response.– Decontextualized, individualized format of turnkey Decontextualized, individualized format of turnkey

systems lend nicely to random assignment.systems lend nicely to random assignment. Innovators run risk of being labeled “unproven”.Innovators run risk of being labeled “unproven”.

Addressing Curricular Tensions Addressing Curricular Tensions with Assessment & Alignmentwith Assessment & Alignment

Good assessment promotes coverage Good assessment promotes coverage and and inquiry.inquiry. ““Learner-oriented” formative feedback:Learner-oriented” formative feedback:

– Attaches new content to robust inquiry schema.Attaches new content to robust inquiry schema.– Promotes participation in discourse and argumentation.Promotes participation in discourse and argumentation.

Good classroom accountability:Good classroom accountability:– Guides student and teacher curricular enactments.Guides student and teacher curricular enactments.– Guides refinement of materials and methods.Guides refinement of materials and methods.– Affords shared knowledge base for subsequent inquiry. Affords shared knowledge base for subsequent inquiry. – Provides additional incentives to motivate engagement.Provides additional incentives to motivate engagement.

Design-based refinements of activities and Design-based refinements of activities and classroom assessments ensures needed outcomes.classroom assessments ensures needed outcomes.

Functions of AssessmentFunctions of Assessment

Almost all assessment serves formative goals:Almost all assessment serves formative goals:– Ultimate goal is improving teaching & learning.Ultimate goal is improving teaching & learning.

More summative functions get all the attention:More summative functions get all the attention:– Summative functions undermine formative.Summative functions undermine formative.

Range of formative functions:Range of formative functions:– Guiding curricular selection.Guiding curricular selection.

– Guiding curricular revision.Guiding curricular revision.

– Guiding curricular enactment.Guiding curricular enactment.

– Directly advancing student understanding.Directly advancing student understanding.

Reflective Assessment/Alignment Reflective Assessment/Alignment FrameworkFramework

Refined in studies of technology-based Refined in studies of technology-based innovations: innovations: – CTGV, CTGV, GenScope, LBD, VSS, GenScope, LBD, VSS, NASA COTF.NASA COTF.

Aims to increase evidential (summative) Aims to increase evidential (summative) and consequential (formative) validity.and consequential (formative) validity.

Considers Considers levels levels of assessment:of assessment:– Activity, Curriculum, & Standards-OrientedActivity, Curriculum, & Standards-Oriented

Considers Considers approaches approaches to assessment:to assessment:– Empiricist, Rationalist, Sociocultural.Empiricist, Rationalist, Sociocultural.

Stanford’s Five Levels of Stanford’s Five Levels of Summative TestingSummative Testing

Immediate: Immediate: – artifacts from the enactment of the curriculum;artifacts from the enactment of the curriculum;

Close:Close:– parallel to the content of a unit; parallel to the content of a unit;

Proximal:Proximal:– Tapping the knowledge and skills of a curricula but not Tapping the knowledge and skills of a curricula but not

necessarily the topics. necessarily the topics. Distal.Distal.

– Reflecting state and national standards;Reflecting state and national standards; Remote:Remote:

– Broader outcomes (e.g., NAEP, workforce) Broader outcomes (e.g., NAEP, workforce)

Levels of AssessmentLevels of Assessment

Embedded (Embedded (Immediate Immediate ):):– Is the curriculum.Is the curriculum.– Informal, collaborative, minimally summativeInformal, collaborative, minimally summative

Activity-Oriented (Activity-Oriented (CloseClose):):– Somewhat formal, individually completed.Somewhat formal, individually completed.– Modestly summative.Modestly summative.

Curriculum-Oriented Curriculum-Oriented (Proximal(Proximal):):– Sensitive to specific curriculum and standards.Sensitive to specific curriculum and standards.– More formal, fairly summative.More formal, fairly summative.

Standards-Oriented (Standards-Oriented (DistalDistal):):– Reflects broader standards.Reflects broader standards.– Extremely formal and highly summative.Extremely formal and highly summative.

Close Assessment:Close Assessment:Activity-Oriented “Quizzes”Activity-Oriented “Quizzes”

Similar context, content, and representation: Similar context, content, and representation: – Screen captures or paper versions of physical lab.Screen captures or paper versions of physical lab.– Use multiple-choice only when appropriate.Use multiple-choice only when appropriate.– Short answer + explanation is a useful format.Short answer + explanation is a useful format.

Ideal for guiding, tuning & revising activities.Ideal for guiding, tuning & revising activities.– Completed at the end of a unit (e.g., weekly).Completed at the end of a unit (e.g., weekly).– Extremely sensitive to enacted activities.Extremely sensitive to enacted activities.– Should directly guide enactment, support self-assessment.Should directly guide enactment, support self-assessment.

Ideal for directly furthering student knowledge.Ideal for directly furthering student knowledge.– Need “learner-oriented” feedback rubrics.Need “learner-oriented” feedback rubrics.– Coach participation in “assessment conversation”.Coach participation in “assessment conversation”.– Don’t let summative function undermine formative goals.Don’t let summative function undermine formative goals.

Proximal Assessment:Proximal Assessment:Curriculum-Oriented “Exams”Curriculum-Oriented “Exams”

Cherry picked or custom items and tasks.Cherry picked or custom items and tasks.– Same content, but new and formal context.Same content, but new and formal context.– Sensitive to curriculum Sensitive to curriculum and and standards.standards.– Use PALS, NAEP, released forms, text supplement.Use PALS, NAEP, released forms, text supplement.

Ideal for assessing & refining curricular coverage Ideal for assessing & refining curricular coverage – Use to guide refinement of activities and quizzes. Use to guide refinement of activities and quizzes. – Good for classroom accountability (grades).Good for classroom accountability (grades).

Ideal for comparing enactments of innovation.Ideal for comparing enactments of innovation.– Can’t give to comparison groups.Can’t give to comparison groups.

Useful for furthering student knowledge:Useful for furthering student knowledge:– Requires students to formalize knowledge.Requires students to formalize knowledge.– Need summative Need summative andand formative rubrics. formative rubrics.

Distal Assessment:Distal Assessment:Standards-Oriented “Tests”Standards-Oriented “Tests”

Randomly-selected items from larger pool:Randomly-selected items from larger pool:– New context and some new content.New context and some new content.– Reflects broader standards.Reflects broader standards.– Same methods used for high-stakes tests.Same methods used for high-stakes tests.

Ideal for summative evaluation and Ideal for summative evaluation and curricular comparisons.curricular comparisons.– Test must be protected from compromise.Test must be protected from compromise.– Different items on pretest and posttest.Different items on pretest and posttest.– Provide only score-level feedback.Provide only score-level feedback.

Approaches to AssessmentApproaches to Assessment

Assessment practices reflect assumptions Assessment practices reflect assumptions about knowledge:about knowledge:– Assumptions are often implicit.Assumptions are often implicit.

Competing assumptions = competing tests:Competing assumptions = competing tests:– Supports validity of high-stakes tests.Supports validity of high-stakes tests.– Explains how innovators compromise validity.Explains how innovators compromise validity.

Generally 3-4 sets of assumptions:Generally 3-4 sets of assumptions:– ““Behavioral, Cognitive, & Cultural”.Behavioral, Cognitive, & Cultural”.– ““Familiar, Important & Enduring”.Familiar, Important & Enduring”.

Three Approaches to AssessmentThree Approaches to Assessment Behavioral/Empiricist focus on lower-level associations:Behavioral/Empiricist focus on lower-level associations:

– Recognition and recall of specific associations.Recognition and recall of specific associations.– Supports multiple-choice standardized tests.Supports multiple-choice standardized tests.

Cognitive/Rationalist focus on higher-level conceptual Cognitive/Rationalist focus on higher-level conceptual schema.schema.– Problem-solving focus on rationalization.Problem-solving focus on rationalization.– Central to 90’s assessment reformCentral to 90’s assessment reform

Situative/Sociohistoric focus on participation in Situative/Sociohistoric focus on participation in knowledge practices.knowledge practices.– Shift from acquisition to participation.Shift from acquisition to participation.– Shift from individual to event.Shift from individual to event.– Reconcile tensions between two acquisitory approachesReconcile tensions between two acquisitory approaches

Assessment Levels x Assessment ApproachesAssessment Levels x Assessment Approaches

ASSESSMENT LEVELASSESSMENT LEVEL

(Primary function, (Primary function, context of use).context of use).

ASPECT OF PROFICIENCYASPECT OF PROFICIENCY

(Conceptualization of knowledge transfer) (Conceptualization of knowledge transfer)

A. BehavioralA. Behavioral(recognize & recall)(recognize & recall)

B. CognitiveB. Cognitive(rationalize)(rationalize)

C. CulturalC. Cultural(ritualize)(ritualize)

1. ACTIVITY-1. ACTIVITY-ORIENTED QUIZZES ORIENTED QUIZZES (after activity, ideal for (after activity, ideal for directly advancing directly advancing knowledge)knowledge)

1A. MC or SA 1A. MC or SA items created for items created for activity.activity.

1B. PA or essay 1B. PA or essay items, same context items, same context as activityas activity

1C. Enactment 1C. Enactment of intended of intended routines, relative routines, relative to ideals.to ideals.

2. CURRICULUM-2. CURRICULUM-ORIENTED EXAMS ORIENTED EXAMS (after curriculum, ideal (after curriculum, ideal for refining curriculum)for refining curriculum)

2A. MC or SA 2A. MC or SA items created or items created or selected to match selected to match curricular targets.curricular targets.

2B. PA or essay 2B. PA or essay items, somewhat items, somewhat different context different context from curriculum.from curriculum.

2C. Discourse 2C. Discourse during somewhat during somewhat different different problems. problems.

3. STANDARDS-3. STANDARDS-ORIENTED TESTSORIENTED TESTS (pre and post, in (pre and post, in innovative and innovative and comparison curriculum)comparison curriculum)

3A. Externally 3A. Externally developed MC or SA developed MC or SA items, aligned to items, aligned to standards, randomly standards, randomly selected.selected.

3B. Externally 3B. Externally developed PA or developed PA or essay items, very essay items, very different context different context from curriculum.from curriculum.

3C. Discourse 3C. Discourse while completing while completing very different very different problems.problems.

Suggestions for CCMS Suggestions for CCMS AlignmentAlignment

Clarify functions of assessment:Clarify functions of assessment:– Don’t confound formative and summative.Don’t confound formative and summative.– Separate activity-oriented and standards-oriented Separate activity-oriented and standards-oriented

assessments.assessments.– Maximize consequential validity of formative.Maximize consequential validity of formative.– Maximize evidential validity of summative.Maximize evidential validity of summative.

Discriminate between approaches to assessment.Discriminate between approaches to assessment.– Greeno, Collins, & Resnick; Hickey & ZuikerGreeno, Collins, & Resnick; Hickey & Zuiker– Wiggins’ Wiggins’ familiar, important, & enduring.familiar, important, & enduring.

Exploit incentive value of summative functions.Exploit incentive value of summative functions. Use IRT (Use IRT (Facets Facets or or WinMiraWinMira) whenever possible.) whenever possible.

Suggestions for CCMS FeedbackSuggestions for CCMS Feedback

Exploit the formative value of assessment:Exploit the formative value of assessment:– Directly advance student understanding.Directly advance student understanding.– Guide teacher’s enactment of new activity.Guide teacher’s enactment of new activity.– Guide students’ enactment of later activity.Guide students’ enactment of later activity.

Develop good classroom assessments:Develop good classroom assessments:– Create learner-oriented rubrics for each Create learner-oriented rubrics for each

assessment (“answer explanations”).assessment (“answer explanations”).– Include new content.Include new content.– Coach assessment conversation.Coach assessment conversation.

General SuggestionsGeneral Suggestions

Use design-based methods (Sloane & Use design-based methods (Sloane & Gorard, 2003) for iterative refinement:Gorard, 2003) for iterative refinement:– Use quizzes to refine activities;Use quizzes to refine activities;– Use exams to refine quizzes and feedback;Use exams to refine quizzes and feedback;– Use tests to refine curriculum, exams, and exam Use tests to refine curriculum, exams, and exam

feedback.feedback.– Build local-level prototheory along the way.Build local-level prototheory along the way.

Include a more comprehensive view of Include a more comprehensive view of assessment in 2061 criteria.assessment in 2061 criteria.

Assessment Levels x Assessment ApproachesAssessment Levels x Assessment Approaches

ASSESSMENT LEVELASSESSMENT LEVEL

(Primary function, (Primary function, context of use).context of use).

ASPECT OF PROFICIENCYASPECT OF PROFICIENCY

(Conceptualization of knowledge transfer) (Conceptualization of knowledge transfer)

A. BehavioralA. Behavioral(recognize & recall)(recognize & recall)

B. CognitiveB. Cognitive(rationalize)(rationalize)

C. CulturalC. Cultural(ritualize)(ritualize)

1. ACTIVITY-1. ACTIVITY-ORIENTED QUIZZES ORIENTED QUIZZES (after activity, ideal for (after activity, ideal for directly advancing directly advancing knowledge)knowledge)

1A. MC or SA 1A. MC or SA items created for items created for activity.activity.

1B. PA or essay 1B. PA or essay items, same context items, same context as activityas activity

1C. Enactment 1C. Enactment of intended of intended routines, relative routines, relative to ideals.to ideals.

2. CURRICULUM-2. CURRICULUM-ORIENTED EXAMS ORIENTED EXAMS (after curriculum, ideal (after curriculum, ideal for refining curriculum)for refining curriculum)

2A. MC or SA 2A. MC or SA items created or items created or selected to match selected to match curricular targets.curricular targets.

2B. PA or essay 2B. PA or essay items, somewhat items, somewhat different context different context from curriculum.from curriculum.

2C. Discourse 2C. Discourse during somewhat during somewhat different different problems. problems.

3. STANDARDS-3. STANDARDS-ORIENTED TESTSORIENTED TESTS (pre and post, in (pre and post, in innovative and innovative and comparison curriculum)comparison curriculum)

3A. Externally 3A. Externally developed MC or SA developed MC or SA items, aligned to items, aligned to standards, randomly standards, randomly selected.selected.

3B. Externally 3B. Externally developed PA or developed PA or essay items, very essay items, very different context different context from curriculum.from curriculum.

3C. Discourse 3C. Discourse while completing while completing very different very different problems.problems.

GenScope Project ResultsGenScope Project ResultsNear-Transfer Gains for Mr. N by Year

20

25

30

35

40

45

50

Pre Post

New

Wor

m P

rofi

cien

cy .

(T-S

cale

)

Mr. N Year 1 GenScope (3, 66)

Mr. N, Year 2 GenScope (4, 68)

Mr. N, Year 3 GenScope (4, 53)

Non-GenScope Comp. (2, 16)

GenScope Project ResultsGenScope Project Results

Far-Transfer Gains for Mr. N by Year

30

35

40

45

50

Pre Post

Mul

tiple

Cho

ice

Pro

fici

ency

.(T

-Sca

le)

Mr. N Year 1 GenScope (3, 66)

Mr. N, Year 2 GenScope (4, 68)

Mr. N, Year 3 GenScope (4, 53)

Non-GenScope Comp. (2, 16)