Post on 06-Nov-2021
PROSEDUR
OSCE, MINI-CEX DAN PORTOFOLIO
Susanto Nugroho
Laboratorium Ilmu Kesehatan Anak
Fakultas Kedokteran Universitas Brawijaya
Sir William Osler (1849-1919)
Father of Modern Medicine
“He who studies medicine without
books sails an uncharted sea, but he
who studies medicine without
patients does not go to sea at all”
“No teaching without the patient for
a text and the best teaching is often
that taught by the patient himself”
…....in Clinical Teaching/Training
How should we know if students
competence?How should we know if
students have
good performance?
“CLINICAL ASSESSMENT”
Patient is very IMPORTANT !!!
OBJECTIVE
After this lecture, all participants will be able
to describe and apply these clinical
assessments following in medical
education/training:
• Objective Structured Clinical
Examination (OSCE)
• Mini-Clinical Evaluation Exercise (Mini-
CEX)
• Portfolio
Why should we
assess the medical
students in clinics ?
Clinical
Assessment
• Evidence of competence/performance &
inform progression (Tomorrow’s doctors -
GMC, 2003)
• To drive learning (van der Vleuten, 2000)
• To improve trainee confidence (van der
Vleuten, 2000)
• Public confidence:
- Scepticism of profession to self-
regulate (Smith, 1998)
- Better measures of quality of practice (Scally, 1998)
• To rebuke legal challenges (Tweed and
Why to assess the medical
students ?
What should we
assess the medical
students in clinics ?
Clinical
Assessment
COMPETENCE
~ “able to do”
PERFORMANCE
~ “actually does”
Clinical
Assessment
How should we
assess the medical
students in clinics ?
Clinical
Assessment
• OSCE ~ Objective structure clinical examination
• OSLER ~ Objective structured long case
examination record
• OSATS ~ Objective structures assessment of
technical skills
How to assess the clinical
competency?
...... undertaken ”outside the real” clinical environment
...... have many aspects of realism of workplace
How to assess the clinical
performance?• Mini-CEX ~ Mini-clinical evaluation exercise
• DOPS ~ Direct observation of procedural skill
• CbD ~ Case-based discussion
• Mini-PAT ~ Mini peer assessment tool
...... undertaken ”on the real” patient & workplace
...... have many aspects of realism of workplace
Assessment ~ Miller’s Pyramid (Miller, 1990)
Does
Shows how
Knows how
Knows
Knowledge
Performance
A framework for assessing clinical competence and performance
Mini-CEX, DOPS, CbD,
Mini-PAT, Portfolio
OSCE, short case,
long case
MEQ, EMQ, PMPs,
SAQ, SEQ
MCQ, Essay, Oral
Competence
Does
Shows how
Knows how
Knows
Knowledge
Performance
A framework for assessing clinical competence and performance
OSCE
Competence
Assessment ~ Miller’s Pyramid (Miller, 1990)
UU Praktek Kedokteran Konsil Kedokteran
Indonesia (KKI) SKDI & Standar Pendidikan
Profesi Dokter pada tahun 2006
(Direvisi tahun 2012)
KBK
Kemampuan
Institusi Berbeda
Kualitas
BerbedaSTANDARISASI
Latar Belakang
Standar Pendidikan Dokter
INPUT PROSES OUTPUT OUTCOME
Kualitas
penerimaan
Kualitas
pendidikan &
pembelajaran
Kualitas
lulusanKualitas
profesionalisme
ASESSMENT
OSCE
Affandi, 2008
Objective: semua peserta diuji dengan
ujian yang samaO
SStructured: penilaian di setiap stasion
terstruktur & yang diujikan adalah
ketrampilan klinik tertentu (anamnesis,
PF, prosedur tindakan, dll)
CEClinical Examination: penilaian
terhadap kemampuan ketrampilan klinik
(bukan pengetahuan) & mahasiswa
harus mendemonstrasikan
Pengertian OSCE
Tujuan OSCE
Menilai kompetensi dan ketrampilan klinis
mahasiswa secara objektif dan terstruktur.
Komponen dalam Pelaksanaan
OSCE• Blue print & soal
ujian
• Penguji
• Pasien standar
(PS)
• Pelatih PS
• Peserta/kandidat
• Manekin & peralatan
ujian
• Sarana & prasarana
• Supporting team/staf
Rancangan OSCE
Station 1
Station 2
Station 3
Station 4
Station 5
Station 6
Station 7
Station 8
Station 9
Station 10
Station 11
Station 12
Rancangan OSCE
Heteroanamnesisibu dengan anak
sakit
PemeriksaanAbdomen
PemeriksaanRefleks
Anamnesis Penyakit
Kronis
PemeriksaanGinekologis
Bedah Minor
MelakukanProsedurAseptik
Blue Print
Blue Print OSCE
• Kategori kompetensi: kemampuan
anamnesis, pemeriksaan fisik, penunjang &
interpretasinya, prosedur tindakan, edukasi
& profesionalisme sesuai standar
kompetensi
• Sistem organ/divisi ~ sesuai dengan yang
akan diujikan
• Kasus: untuk memberikan situasi klinik
yang diharapkan
• Distribusi & proporsi pencapaian
kompetensi: perilaku profesional harus
100%
Katagori Kompetensi
Min
ima
l
Sis
tem
Muskulo
skele
tal
Sis
tem
Muskulo
skele
tal
Sis
tem
Muskulo
skele
tal
Sis
tem
Hem
ato
poie
tk
Sis
tem
Hem
ato
poie
tk
Sis
tem
Kulit
&
jaringan
Ikat
Sis
tem
Kulit
&
jaringan
Ikat
Sis
tem
En
do
kri
n&
Meta
bo
lik
Kasus
1. Anamnesis 1 +
2. Pemeriksaan fisik 1 + +
3. Melakukan
tes/prosedur
klinik/interpretasi
data
3 + + +
4. Menentukan
diagnosis atau
diagnosis banding
5 + + + + +
5. Penatalaksanaan:
a. Non
Farmakoterapi
1 +
b. Farmakoterapi 4 + + + +
6. Komunikasi &
Template Stasion OSCE
Standard Setting OSCE
Absolute Methods
• Anggoff (modified)
• Ebel
Compromise Methods
• The Hofstee
Method
• Borderline Group
Method
• Borderline
Regression
Method
Borderline Regression Method
• Metode standar setting yang sering digunakan
pada OSCE
• Penilaian meliputi: “Actual Mark” dan “Global
Rating”
• Actual Mark: deskripsi skor (0 s/d 3) di daftar
tilik (rubrik) harus jelas agar penguji tepat
dalam memberikan skor
• Global Rating: persepsi penguji terhadap
“overall performance” (meminimalisasi
subyektivitas penilaian)
1 = tidak lulus
2 = borderline (minimally competence)
3 = lulus
Penilaian (Skor) yang
DimasukkanTotal nilai :
Actual Mark : ..............
Global Rating :
1. tidak lulus 2. borderline
3. lulus 4. outstanding
Penentuan “Minimum Passing
Level ” (MPL)
10
5
15
20
25
30
1 2 3 4
NO GR AM
1 1 10
2 3 20
3 2 14
4 2 18
5 3 22
6 4 28
7 4 30
8 3 26
9 2 16
10 3 24
Global Rating
Actual Marck
1 4 242 1 123 1 94 2 155 2 206 3 227 2 168 3 179 1 10
10 2 1211 3 1412 4 2213 4 2114 3 1615 1 816 4 2617 3 2018 2 1119 3 1620 1 8
0
5
10
15
20
25
30
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
Act
ual
Mar
ck
Global Rating
Actual Marck
Linear (Actual Marck)
14
≥14
Lulus
Kelebihan OSCE
1. Valid
2. Reliabel
3. Setting klinik yang nyata dan menarik
4. Berbagai ketrampilan dengan variasi dapat
diujikan dalam waktu yang relatif singkat
5. Setting standar kompetensi tertentu dapat
ditentukan
6. Obyektif: variasi pasien dan penguji dikurangi
7. Format OSCE bersifat fleksibel
8. Pengamatan langsung pada setiap peserta
9. Terstruktur dan terencana
10.Feasibel
Kekurangan OSCE
1. Kompartementalisasi
2. “High cost”
3. “High human resources”:
- Pasien standar (PS)
- Pelatih PS
- Penguji yang terlatih: perlu pelatihan
4. “High time consuming”: untuk persiapan,
pelaksanaan dan evaluasi
5. Perlu organisasi dan koordinasi yang baik
Does
Shows how
Knows how
Knows
Knowledge
Performance
A framework for assessing clinical competence and performance
MINI-CEX
Competence
Assessment ~ Miller’s Pyramid (Miller, 1990)
Method to assess clinical performance of students in
workplace – “DOES” level (Miller’s pyramid)
Key features – real patients, clinical conditions, work
settings, clinical tasks and constrains
Relies on – multiple encounters, assessors, settings,
occasions and judgements
Involves – clinicians to score, short focused patient-trainee
interaction, and clinician offering ± 5 minutes of
developmental feedback to trainee to improve quality of
clinical skills
Features of Mini-CEX
Can assess the clinical skills – based on standard of
competencies
Can assess the clinical performance – clinical skills,
student – patient interaction (attitudes & behaviour)
Direct observation & immediate feedback – not only
helps student to identify strength & weakness, but can helps
to improve skills
Need few minutes – 10-15 minutes of observation & 5-10
minutes of feedback
Advantages of Mini-CEX
Mini-CEX Forms
Student’s information
Assessment
Feedback & action plan
Assessor ID
Competencies assessed &
descriptors
Step 1. Preparation
Step 2. Observation &
Assessment
Step 3. Verbal & Written
Feedback
The Validity of Mini-CEX
The Reliability of Mini-CEX
The mini-CEX is a reliable tool for performance
assessment, and is acceptable to and well received
by both learners and supervisors.
Nair, et al., 2008
Changes in reliability as a function of
the number of encounters
Norcini & Burch, 2007
Students should be observed at
least four times by different
assessors to get a reliable
assessment of competence.
Mini-CEX is a good example of workplace-
based assessment method that fulfils three
requirements for facilitating learning:
1. The course content, expected competencies &
assessment practices are aligned
2. Feedback is provided either during or
immediately after the assessment
3. The assessment is used to direct learning
towards desired outcomes
The Reliability of Mini-CEX
Norcini & Burch, 2007
The Educational Impacts of
Mini-CEX
Liao, et al., 2013
• Knowledge acquisition and enhancement of
giving feedback when the faculty members
used the tool.
• Providing effective feedback should be
conducted to increase the impact of the mini-
CEX as a formative assessment.
Mini-CEX ~ a valid, reliable & feasible
Validity – it is important instrument for the direct observation
of trainee’s clinical performance
Reliability
• more encounters & assessors - more reliable
• expected competency, feedback & used to direct learning
Educational impacts
• can correct the weakness & mature professionally
• can monitor progress & identify educational needs
• reassures student’s satisfactory performance
• increases the interaction student & teacher
Conclusions
Assessment ~ Miller’s Pyramid
Does
Shows how
Knows how
Knows
Knowledge
Performance
A framework for assessing clinical competence and performance
Portfolio
Competence
Portfolio
EXAMINATIONtoward broader methods of
assessment
• To encourage closer links between assessment &
learning using assessment & feedbackby
learning improvement
• To enhance the assessment of areas that are
difficult to assess by traditional methods: attitudes,
personal attributes, reflection & professionalismHaldane, 2014
A purposeful collection of work(Stecher, 2001)
A collection of papers and other
forms of evidence that learning has
taken place
(Davis et al.,
2001)A collection of student work that exhibits the
student’s efforts, progress and achievements in
one or more areas(Gisselle & Martin-Kniep, 2000)
Definition of Portfolio
1. Portfolios’ contribution to assessment
2. Focus on personal attributes
3. Enhances interactions between students &
teachers
4. Stimulates the use of reflective strategies
5. Expands understanding of professional
competence
Why Use Portfolio ?
Models of Portfolio
Portfolio Description Advantages Disadvantages
Shopping
trolley
Contains anything which
has been produced or
used during the learning
proces
Very inclusive Difficult to assess.
No analysis of
contents
Toast rack “Toast” for each period of
learning
Corresponds with
the curriculum
Can be marked
Includes
reflection
Each item is
discrete & does not
provide overall
assessment of
learning
No overall
reflection
Cake mix Integration of the parts
“Mixing” is reflection on
the
analytical components
Global
assessment
Individual
components may
not be clear
Spinal
column
Series of competency
statements are the
Each
competency has
• Portfolio model should be easily aligned with
the curriculum ~ “toast rack” or “spinal
column”
• Portfolio should provide opportunity to
demonstrate learning in many different ways
& should be a holistic record of learning
• The appropriate model depends on its
purpose will be used in assessment
process
• Structure should be decided based on the
format the evidence is required
Portfolio Contents, Alignment &
Assessment
Haldane, 2014
Five Steps in Portfolio
Assessment Process
Davis & Ponnamperuma, 2006
Step
I.
EvidenceDocumentation of experience by the learner
Step
II.
Reflection Commentary by the learner on experiences
and learning that has resulted
Step
III.
EvaluationStudying the evidence by examiners
Step
IV.
Defending the evidenceA dialogue between learner and examiner
Step
V.
Assessment decisionFormative and summative
Learning Outcome
A three-circle classification model of
learning outcome
A. Inner circle ~ “what the doctor is able to
do”
(1) Clinical skills
(2) Practical procedures
(3) Investigating a patient
(4) Patient management
(5) Health promotion & disease prevention
(6) Communication skills
(7) Information handling & retrieval Harden et al, 1999
B. Middle circle ~ “how the doctor
approach the task”
(8) Understanding of basic, clinical & social
sciences
(9) Appropriate attitudes, ethical understanding
& legal responsibilities
(10) Appropriate decision making, clinical
reasoning & judgementC. Outer circle ~ “doctor as a professional”
(11) The role of the doctor within the health
service
(12) Attitude for personal development
Harden et al, 1999
1. Defining the purpose
2. Determining competences to be assessed
3. Selection of portfolio material
4. Developing a marking system
5. Selection and training of examiners
6. Planning the examination process
7. Student orientation
8. Developing guidelines for decisions
9. Establishing reliability and validity evidence
10. Designing evaluation procedures
Steps in Developing Portfolio
AMEE Medical Education Guide, 2001
1. Defining the purpose
The Purposes of Portfolio
1. For the trainee:
• to record the training experience, education
supervision, professional development
plans, workshops attended, reflective entries
and assessment reports
• to identify deficiencies
• to plan for necessary remediation
2. For supervisors:
• to assess overall training & work with the
trainee
• to correct deficiencies
Joint Committee on Specialist Training, 2011
Expected achievement in the 6 core
competencies:
1. Patient care
2. Medical knowledge
3. Practice based learning & improvement
4. Interpersonal & communication skills
5. Professionalism
6. System based practice
Expected Achievement of
Competencies
2. Determining competences to be
assessed
Joint Committee on Specialist Training, 2011
Expected frequency of assessments
Other areas of curriculum & assessment
3. Selection of portfolio material
To assess: patient management skills, e.g.
patient education
• Written outline of a patient education
programme in community
• A video of individual patient education
session with patient discussing one topic
To assess: attitudes, ethical & legal
understanding & responsibility
• The elective report ~ the student shows
ethical understanding of issues inherent in the
elective
• Case discussion on ethics ~ the students will
provide evidence of ethical judgement & moral
reasoning & could be questioned about the
Record the cases
Reflective entries
DOPS for
Medical
Training
Record of Procedures
Name of procedure: Intraosseous needle insertion
Name of procedure: Percutaneous central line insertion
Practice Based Learning Assessment
Teaching of Communication Skill Evaluation
4. Developing a marking system
• Students’ work is judged by criteria which will
specify the level of their academic
achievements and will determine their progress
towards state standards.
• The portfolio material should direct the
examiner to consider student progress
according to the outcome specification & should
enable the examiner to identify strengths &
weaknesses.
• If one competence is assessed, highly specific
criteria could be employed; if multiple
competences, general standards should be
developed.
5. Selection & training of examiners
• According to its purpose of portfolio the
appropriate examiners, include: staffs,
teachers in the basic sciences & laboratory-
based diciplines, clinicians, faculty who
indicate special interest in education &
student development.
• Another selection issue: seniority of
examiners
• A key point of the success of programme:
the training of faculty examiners &
maintaining them should be preserved &
reinforced
6. Planning
the
examination
process
7. Student orientation
• Students must be informed at the beginning
of the course about the portfolio
examination, the guidelines & criteria for
judging performance clearly.
• Students can use their on-going work for
selection of material (if they demonstrate
good progress in their achievements & have
confidence in ability to pass the portfolio)
• The more information given to students the
more positive they become towards the
portfolio.
8. Developing guidelines for
decisionsFlow diagram for the decision-making process
9. Establishing reliability & validity
evidence• It is important to determine what will
constitute good reliable evidence & plan the
examination, e.g two independent
examiners, one examiners, independent
rating, consensus or both, minimum desired
reliability or generalisability co-efficient.
• Define desired correlations or absolute inter-
rater agreement and set the minimum
standards for tolerance of misclassification
error.
• Triangulation of portfolio results with other
forms of assessment will increase the
validity of the decision.
10. Designing evaluation
procedures Feedback: “student” & “examiners” opinion on the
portfolio’s strength & weakness changes &
improvements
• Questionnaires
• Focus group discussion
• Individual interview
• Request for written comments
Portfolio’s Assessment Features
1. Formative & summative
2. Qualitative & quantitative
3. Personalised
4. Standarised
5. Authentic
Thank You for Your Attention