Authentic Performance Assessment
-
Upload
myra-charles -
Category
Documents
-
view
44 -
download
3
description
Transcript of Authentic Performance Assessment
Slide 1
Authentic Authentic Performance Performance AssessmentAssessment
Ellen J. Lehning, Ph.D.Ellen J. Lehning, Ph.D.Department of AnesthesiologyDepartment of Anesthesiology
Slide 2
ObjectivesObjectives
Define clinical competence and Define clinical competence and authentic performanceauthentic performance
Design an authentic performance Design an authentic performance assessmentassessment
Implement an authentic Implement an authentic performance assessmentperformance assessment
Slide 3
Teach to and Test Desired Teach to and Test Desired Endpoints:Endpoints:
What is the First Principle of What is the First Principle of Education?Education?
Educational OutcomesEducational Outcomes
Slide 4
Pass Oral BoardsPass Oral Boards
Pass Written Pass Written BoardsBoards
Achieve Clinical Achieve Clinical CompetenceCompetence
Desired Outcome: Desired Outcome: Assess: Assess:
Slide 5
What is Clinical Competence?What is Clinical Competence?
““IIntegrationntegration of knowledge, skills of knowledge, skills and attitudes.”and attitudes.”
Rice et al., Competency-based objectives for clinical training. Can. J. Med. Technol. 57:136, 1995
Slide 6
What is Clinical Competence?What is Clinical Competence?
““The habitual and judicious use The habitual and judicious use of communication, knowledge, of communication, knowledge,
technical skills, clinical technical skills, clinical reasoning, emotions, values, reasoning, emotions, values,
and reflection in daily practice and reflection in daily practice for the benefit of the individual for the benefit of the individual and community being servedand community being served.”.”
Epstein and Hundert, Designing and assessing professional competence. JAMA 287:226, 2002
Slide 7
What is Clinical Competence?What is Clinical Competence?
Miller, The assessment of clinical skills/competencies/performance. Acad. Med. 65:563-567, 1990.
Knows
Knows how
Shows how
Does
Miller’s Pyramid of Competence
Slide 8
What is Clinical Competence?What is Clinical Competence?
““Competence . . . means being Competence . . . means being able to function in able to function in contextcontext.”.”
Chambers and Gerrow, Manual for Developing and Formatting Competency Statements. J. Dental Educ. 58:361, 1994.
Slide 9
How is Clinical Competence How is Clinical Competence Assessed?Assessed?
““Competence is concerned with what Competence is concerned with what people people cancan do . . ., that is, potential.” do . . ., that is, potential.”
While, Competence versus performance: which is more important? J. Adv. Nurs. 20:525, 1994.
““Performance is concerned with what Performance is concerned with what people people dodo . . ., that is, what is actually . . ., that is, what is actually
done in the real-life context.”done in the real-life context.”
Competence is inferred from performance!Competence is inferred from performance!
Slide 10
What is Authentic Performance What is Authentic Performance Assessment?Assessment?
Performance can be Performance can be simulatedsimulated or or authenticauthentic
SimulatedSimulated performance assessment uses performance assessment uses an an artificialartificial context: context:
SPs, OSCEs, Long Case - Standardized or SPs, OSCEs, Long Case - Standardized or UnstandardizedUnstandardized
AuthenticAuthentic performance assessment uses performance assessment uses a a realreal context: context:
Direct observation of a resident caring for Direct observation of a resident caring for a real patient with a real problem in a a real patient with a real problem in a
real setting – Unstandardized!real setting – Unstandardized!
Slide 11
Why Use Authentic Why Use Authentic Performance?Performance?““Testing should be as close as Testing should be as close as
possible to the situation in which possible to the situation in which one attacks the problem.”one attacks the problem.”
““Ill-structured problems are not found Ill-structured problems are not found in simulated and/or standardized in simulated and/or standardized
tests.”tests.”
““The variation inherent in The variation inherent in professional practice will always professional practice will always elude capture by a set of rules.”elude capture by a set of rules.”
Wiggins, Assessing Student Performance: Exploring the Purpose and Limits of Testing, Jossey-Bass, Inc. 1993
Slide 12
How is it Done?How is it Done?
““There are few validated strategies to There are few validated strategies to assess actual clinical practice.”assess actual clinical practice.”
““Assessment at the apex of Miller’s Assessment at the apex of Miller’s pyramid, the does, is the international pyramid, the does, is the international
challenge of the century for all challenge of the century for all involved in clinical competence involved in clinical competence
testing.”testing.”
- Ronald Epstein
- Val Wass
Slide 13
Design Step 1 – Select Design Step 1 – Select CompetenciesCompetencies
Breakdown ACGME’s six global Breakdown ACGME’s six global competencies into specific competenciescompetencies into specific competencies
Technical skills, case management Technical skills, case management skills, clinical decision-making skills, skills, clinical decision-making skills, etc.etc.
Avoid too broadAvoid too broad
Avoid too specificAvoid too specific
Appropriate for the resident’s training Appropriate for the resident’s training levellevel
Sequenced appropriately over the Sequenced appropriately over the training periodtraining period
Slide 14
Design Step 1 – Select Design Step 1 – Select CompetenciesCompetencies
Write a competency statement:Write a competency statement:
Chambers and Gerrow, Manual for developing and formatting competency statements. J. Dental Educ. 58:361, 1994.
Verb Direct Object Qualifying conditions
Slide 15
Example Competency StatementExample Competency Statement
Induce anesthesia for an ASA I-II Induce anesthesia for an ASA I-II patient by the end of the three patient by the end of the three
month periodmonth period
Slide 16
Example Competency Example Competency StatementsStatements The Global Communication Skills CompetencyThe Global Communication Skills Competency
Communicate accurately, efficiently and supportively Communicate accurately, efficiently and supportively with the patient, patient’s family and the health care with the patient, patient’s family and the health care teamteam
Specific Communication Skill CompetencySpecific Communication Skill Competency
Build a Relationship (Rapport Building)
Component Skills used to Build a Relationship
Establish initial rapportEstablish initial rapport Support SupportEmpathy PartnershipReflection Legitimation
Respect Apology
Slide 17
The Performance Rating ProcessThe Performance Rating Process
Specific performances are judged by expert Specific performances are judged by expert
rater(s) who synthesize multiple impressions in rater(s) who synthesize multiple impressions in
comparison to criteria given in a rating task comparison to criteria given in a rating task
and filtered through the experience of the and filtered through the experience of the
rater.rater.
Chambers, A primer on competency-based evaluation. J. Dent. Educ. 61:651, 1997.
Specific performance(s)Specific performance(s)
JudgmentJudgment
ExpertExpert
Criteria (Criterion referencing)Criteria (Criterion referencing)
Slide 18
Design Step 2 – Select SamplingDesign Step 2 – Select Sampling
Which performances or tasks will be observed?
How many observations?
When?
Slide 19
Design Step 3 – Select RatersDesign Step 3 – Select Raters
Raters are qualified individuals who have an Raters are qualified individuals who have an
opportunity to observe and draw conclusions opportunity to observe and draw conclusions
about residents – they judge the presence and about residents – they judge the presence and
the quality of the competency to be assessedthe quality of the competency to be assessed
How many ratersHow many raters
AttendingsAttendings
Other professional staffOther professional staff
PatientsPatients
SelfSelf
Slide 20
Design Step 4 – Select Rating Design Step 4 – Select Rating CriteriaCriteria
Rating criteria can either be Rating criteria can either be checklistschecklists or or rubricsrubrics
ChecklistsChecklists Presence or absence of a competency or Presence or absence of a competency or
of the components of a competencyof the components of a competency Do not judge qualityDo not judge quality May be appropriate for technical May be appropriate for technical
procedures or beginning level procedures or beginning level
competenciescompetencies May not capture sophistication of May not capture sophistication of
complex performances:complex performances:Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., and Hodges, B., Regehr, G., McNaughton, N., Tiberius, R., and Hanson, M. 1999. OSCE checklists do not capture increasing Hanson, M. 1999. OSCE checklists do not capture increasing levels of expertise. Acad. Med. 74:1129-1134.levels of expertise. Acad. Med. 74:1129-1134.
Slide 21
Example ChecklistExample Checklist
Yes NoCompetency:Built a relationship?
Specific Skills:Established initial rapport? Employed Legitimation? Provided support? Demonstrated empathy? Demonstrated legitimation? Demonstrated apology? Showed respect?
Slide 22
Design Step 4 – Select Rating Design Step 4 – Select Rating CriteriaCriteria
Rubrics Rubrics Guidelines, rules, or principles by Guidelines, rules, or principles by
which performances are judgedwhich performances are judged Reflect best thinking as to what Reflect best thinking as to what
constitutes a good performanceconstitutes a good performance Contain multiple quality levelsContain multiple quality levels All rating levels must be written-down, All rating levels must be written-down,
defined and describeddefined and described Best rubrics contain anchor(s) to Best rubrics contain anchor(s) to
illustrate the different points on the illustrate the different points on the
quality scalequality scaleArter & McTighe, Scoring Rubrics in the Classroom, Corwin Press, Inc. 2001
Slide 23
Example RubricExample RubricExcellent – Smooth and efficient. Able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure.Competent – Lacks smoothness and efficiency but is able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure.Beginner – Lacks smoothness and efficiency. Able to manage the case but exhibits limited use of personal judgment and responsiveness to the specifics of the patient and operative procedure. Requires some limited coaching or attending intervention.Novice – Can only manage the case with extensive coaching and attending intervention.
Slide 24
What are the Desirable What are the Desirable Characteristics of Performance Characteristics of Performance
Assessment?Assessment?Face validityFace validity
Low CostLow Cost
FeasibleFeasible
Non-reactiveNon-reactive
GeneralizableGeneralizable
ValidValid ReliableReliable
Slide 25
Psychometric IssuesPsychometric Issues Write good rubricsWrite good rubrics
Train the ratersTrain the raters– Only need 1 rater/performanceOnly need 1 rater/performance
Sample over a broad array of cases/patientsSample over a broad array of cases/patients
Our intercase agreements range from 0.2 – 0.9Our intercase agreements range from 0.2 – 0.9
Wass et al., Generalizability in range of the Wass et al., Generalizability in range of the
OSCEOSCE
ACGME preliminary data shows adequate ACGME preliminary data shows adequate
generalizability, validity and reliabilitygeneralizability, validity and reliability
Slide 26
Authentic Performances vs. Authentic Performances vs. Authentic ProductsAuthentic Products
Authentic Performances
Patient Care
Professionalism
Communication Skills
Authentic Products
Patient Care
Ethics
PBLI
Systems-Based
Assessing an:
Authentic Performance
versus the
Product of an Authentic Performance
Slide 27
Why Focus on Authentic Why Focus on Authentic Performance Assessment?Performance Assessment?
ACGME outcomes mandateACGME outcomes mandate
Current assessment is rudimentaryCurrent assessment is rudimentary
AccountabilityAccountability
Murray et al., The accountability of clinical education: its definition and assessment. Med. Educ. 34:871, 2000
Bordage et al., Education in ambulatory settings: Developing valid measures of educational outcomes, and other research priorities. Acad. Med. 73:743, 1998