Universiteit Maastricht

Barcelona, 6 – 9 July 2004

11th Ottawa conference onMedical Education

Universiteit Maastricht

where do we go from here?

Where do we go from here?

Lambert Schuwirth

new challenges in assessmentof clinical competence

Universiteit Maastricht

overview of the presentation

- why this presentation

Where do we go from here?

- developments in competence assessment

- implications for research

- developments in performance assessment

Universiteit Maastricht

overview of the presentation

Where do we go from here?

- why this presentation

- developments in competence assessment

- implications for research

- developments in performance assessment

Universiteit Maastricht

experts vs novices: context vs concept

Where do we go from here?

Fz

Ff

1,25 kg

Universiteit Maastricht

overview of the presentation

Where do we go from here?

- why this presentation

- developments in competence assessment

- implications for research

- developments in performance assessment

Universiteit MaastrichtWhere do we go from here?

simple isn’t it?

test

pass

fail

competent

incompetent

Universiteit MaastrichtWhere do we go from here?

simple, it isn’t

competent incompetent

pass

fail

modern approaches to assessment

Universiteit Maastricht

1 re-evaluate the nature of medical competence

2 shift towards more student-centred assessment

3 include general professional competencies

4 broaden the measurement time frame

Where do we go from here?

1 trait orientated view on medical competence

Universiteit Maastricht

knowledge

instrument

instrument

skills

instrument

instrument

problem solving

instrument

instrument

instrument

attitudes

instrument

instrument

instrument

Where do we go from here?

one instrument for one trait

Universiteit MaastrichtWhere do we go from here?

instruments

traits

A B C D

1

2

3

4

one instrument for one trait

Universiteit MaastrichtWhere do we go from here?

old model of expertise: problems

Universiteit MaastrichtWhere do we go from here?

more variation within than between methods

1 trait orientated view on medical competence

Universiteit Maastricht

knowledge

instrument

instrument

skills

instrument

instrument

problem solving

instrument

instrument

instrument

attitudes

instrument

instrument

instrument

Where do we go from here?

1 trait orientated view on medical competence

Universiteit Maastricht

knowledge

instrument

instrument

skills

instrument

instrument

problem solving

instrument

instrument

instrument

attitudes

instrument

instrument

instrument

Where do we go from here?

instrumentinstrument

old model of expertise: problems

Universiteit MaastrichtWhere do we go from here?

more variation within than between methods

content determines the assessed competence

Universiteit Maastricht

knows

knows how

shows how

does

Miller's pyramid

Where do we go from here?

Universiteit Maastricht

2 shift towards student-centred assessment

- self-assessment

- peer-assessment

- co-assessment

self-evaluation skills

assessing the unmeasurable

assessing the unmeasurable

peer pressure?

peer-evaluation skills

self, peer and co-evaluation skills

peer pressure?

joint responsibility

Where do we go from here?

Universiteit Maastricht

3 include general professional competencies

dealing with tasks

dealing with others

dealing with oneself

planning and time management

information gathering

co-operation

communicating

giving feedback and criticising

self-responsibility

self-evaluation

self-reflection

Where do we go from here?

Universiteit Maastricht

4 broaden the measurement time frame

specific test occasions - specific preparation of students

- inflexibility

longitudinal testing

- unwanted influence on learning

- certain aspects unmeasurable

- no specific preparation

- congruent with flexible curricula

- wanted influence on learning

Where do we go from here?

Universiteit Maastricht

lessons learnt

Where do we go from here?

good assessment draws the whole picture

no single method is a panacea

the content is important, not the format

too much structure does not work

sampling strategies are essential sampling strategies are essential

Universiteit Maastricht

concept and context: subjectivity

Where do we go from here?

experts

LambertMozart

Universiteit MaastrichtWhere do we go from here?

sampling strategies are essential sampling strategies are essential

how do you feel now

how did you feel 5’ ago

how did you feel 10’ ago

how will you feel in 5’

how will you feel in 10’

what does your partner about think how you feel

Lambert Neil Dale

7

7

7

7

7

7

5

5

5

5

5

5

9

9

9

9

9

9

Universiteit Maastricht

lessons learnt

Where do we go from here?

sampling strategies are essential sampling strategies are essential

good assessment shows the whole picture

no single method is a panacea

the content is important, not the format

too much structure does not work

Universiteit Maastricht

overview of the presentation

Where do we go from here?

- why this presentation

- developments in competence assessment

- implications for research

- developments in performance assessment

Universiteit Maastricht

definitions

Where do we go from here?

competencewhat professionals can do in controlled representationsof professional practice, performing at the maximum of their ability

performancewhat professionals actually do in their professional practice

Universiteit MaastrichtWhere do we go from here?

1 clinical expertise

2 communication skills

3 collaboration skills

4 management skills

6 educational skills

7 professional attributes

8 personal health & self care

5 personal development

current approaches: still trait orientated

Universiteit MaastrichtWhere do we go from here?

instrumentschart review

data review

interviews

rating scales

surveys

video

portfolios

incognito simulated patiens

critical incident analyses

Universiteit MaastrichtWhere do we go from here?

but…….

so…….

no direct link between instrument and trait

domains are not discrete entities

trait-orientated approach did not work in competence assessment

modern approaches define competence and performance in terms of professional roles

teacher

Universiteit MaastrichtWhere do we go from here?

patient care

person

professional role-orientated goals

scientist

health care management

Universiteit MaastrichtWhere do we go from here?

1 role – 1 instrument?

instruments

roles

A B C D

1

2

3

4

Universiteit MaastrichtWhere do we go from here?

multi-modal assessment

roles

A B C D

1

2

3

4

sources of information

Universiteit MaastrichtWhere do we go from here?

how: seeing the whole picture

Universiteit MaastrichtWhere do we go from here?

how: approaches

- variation in instruments needed

- performance often involves subjective aspects

-collation of information based on content-equal bits and not on method-equal bits

no flight into structuring, but in sampling

Universiteit MaastrichtWhere do we go from here?

how: organisationplan

purposes

elements of defensibility

quality control measures

pathways

implementation

administration

consequences

assessee

Universiteit MaastrichtWhere do we go from here?

observations

assessment is not mainly a measurement problem but more an (educational) design problem.

instrument is not synonymous with a competence

no single instrument is the panacea, a variety isneeded: an assessment programme

content is more important than format:it is the concept that counts, not the format

Universiteit MaastrichtWhere do we go from here?

suggestionsassessment is not mainly a measurement problem but more an educational designproblem.

- built-in quality control procedures and bodies

- sampling in stead of (over) structuring

- variation of methods, compensation, regulations

- saturation of information approach

- training of students, teachers, examiners, board members

masterplan of the assessment system

Universiteit Maastricht

overview of the presentation

Where do we go from here?

- why this presentation

- developments in competence assessment

- implications for research

- developments in performance assessment

Universiteit MaastrichtWhere do we go from here?

trait orientated

assessment as a psychological testing problem

- highly quantitative methodologies

- reliability of methods

- construct validity

+++

+++

+++

Universiteit MaastrichtWhere do we go from here?

trait orientated

assessment as a instructional design problem

- quantitative methodologies

- reproducibility of methods

- construct validity

- qualititative methodologies

- consequential validity

- research into assessment systems

- direct validity

+

+

+

±

++

+++

++++++

Universiteit MaastrichtWhere do we go from here?

observations

research of assessment systems is more importantnow than research into single methods.

methodology is not synonymous to research question

no single one study/methodology will provide thedefinitive answer, a variety is needed

the methodological concept is important not the specific analysis or method

Universiteit MaastrichtWhere do we go from here?

research agenda

what constitutes good assessment programmes?

which building blocks should be included?

how should information be collated:qualitatively / quantitatively?

assessment as a programme:

who should do what?

Universiteit MaastrichtWhere do we go from here?

research agenda

g-analyses

reproducibility of (portfolio) judgments?

other statistical approaches

sequential testing?

organisational approaches?

Universiteit MaastrichtWhere do we go from here?

Research agenda

how to “score” qualitative judgments?

quality elements of multi-modal assessment

identifiers for profile scores

what influences the thinking of judges?

fraud/plagiarism (internet)

flexible adaptation of teaching to assessment outcomes

Universiteit MaastrichtWhere do we go from here?

research agenda

roles of format, content, scheduling and regulations

consequential validity / educational impact

what influences teacher behaviour

Universiteit MaastrichtWhere do we go from here?

research agenda

consequential validity / educational impact

Universiteit MaastrichtWhere do we go from here?

research agenda

how to change a programme?

research into assessment programmes

stakeholder involvement

how are the instruments combined?

how to use parts in stead of whole instruments?

rules of design

Universiteit MaastrichtWhere do we go from here?

SUMMARYSUMMARY

overview of the presentation

- why this presentation

- developments in competence assessment

- implications for research

- developments in performance assessment

Universiteit MaastrichtWhere do we go from here?

conclusions: in (education), assesment and research

strength comes from collation of results

the method is not synonymous with the concept

no single method is a panacea

translation of concepts into methods is useful

Universiteit MaastrichtWhere do we go from here?

Thank you