Some Implications of ExpertiseResearch for Educational

Assessment

Robert J. Mislevy

University of MarylandNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)

2005 CRESST ConferenceUCLA, Los Angeles, CASeptember 8-9, 2005

The Central Idea Expertise research seeks to understand in

detail the knowledge, representations, development, and contexts of expertise.

Results can ground assessment arguments & task design.

Must recast in terms of purpose, perspective, principles, and structures of assessment.

Especially valuable with complex proficiencies, situations, or performances.

Outline

An example: Architectural design

Implications of expertise research

Assessment arguments

Generative schemas for task design

Conclusion

The Architectural Registration Examination

Architectural design; CAD-like environment.

(ETS: Katz, Bejar, Braun, Hone, Brittingham, Bennett, et al.)

To replace 10-hour hand-drawn design problem

Reflects changing of profession to CAD

Premium on thinking, not drawing

Cognitive task analysis

Planning the firestation site:

A “block diagram” design problem

An Example of a Task Prompt for the ARE

An Illustrative Base Diagram for an ARE Task

A Sample Solution to an ARE Task

Some Results

Differences between novices and experts• Success rate: 98% vs. 88%

• Planning time & execution time

• Patterns of revision involving rework

Assessment Arguments

What complex of knowledge, skills, or other attributes should be assessed?

What behaviors or performances should reveal those constructs?

What tasks or situations should elicit those behaviors?

(Messick, 1994)

Implications for task design (1)

Experts perceive/understand/act via fundamental principles rather than surface features• Chi, Feltovich, & Glaser on physics

In architecture, Constraints• Number • Variation in importance/difficulty• Degree of conflict• Implicit constraints

Implications for task design (2)

Importance of interaction with situation Intelligence built into situations, tools,

processes “Attunement to affordances” Cycles of …

Approach to solution, as in AREHypothesize, test, revise, as in

inquiry & troubleshooting

Implications for task design (3)

External knowledge representationsMaps, insurance forms, symbol systems, diagrams,

Punnett squares, wiring charts, blueprints

Roles in practice Gather/create/share/hold/transform knowledge Embody generative principles of domain Central to practice in the domain

Roles in assessment Environment / stimuli / work of tasks Design KRs to create / implement assessment

CTA for Assessment

CTA can help ground assessment, re validity argument and principled/explicated task construction. Must aim to specify elements of Student-,

Evidence-, and Task-models

in light of the intended assessment’s purpose,

and constraints of the assessment-building and assessment-delivery contexts.

Design patterns

Look across domains to find recurring difficulties, blockages, overloads (Salthouse, 1991)

Classes of expertise / observables / situations that arise in many domains.

Can create “Design patterns” in, e.g.,• Design under constraints (engineering)

• Problem-solving in finite domains (troubleshooting)

• Model-based reasoning (science)

Advances in assessment design

A cognitive design system approach to generating valid tests (Embretson,1998)

Model-basel assessment (Baker, 1997)

Constructing measures (Wilson, 2004)

Understanding by design (Wiggins, 1998)

On the structure of educational assessments (Mislevy, Steinberg, & Almond, 2003)

Conclusion

• Insights from expertise research can improve the practice of assessment.

• Suitable conceptual frameworks, tools, and exemplars are now appearing.

• Design & delivery frameworks are key to making technology-based / complex assessment practical.