SOURCES OF OUTCOME DATA Internal measures –End of course & clerkship surveys –End of year...
-
Upload
ashlee-wiggins -
Category
Documents
-
view
214 -
download
1
Transcript of SOURCES OF OUTCOME DATA Internal measures –End of course & clerkship surveys –End of year...
SOURCES OF OUTCOME DATA
• Internal measures– End of course &
clerkship surveys– End of year surveys– Faculty survey– Universal Student
Rating of Instruction (USRI)
– Certifying exam scores– Canadian Graduate
Questionnaire
• External measures– MCC– Resident program directors’
evaluation of graduates – NBME-Comprehensive
Basic Science Exam– CaRMs– Alumni– LCME report– Alberta
Universities/Colleges Graduate Employment Survey
A Jones, Associate Dean – University of Calgary
Overall Rating of First Year Courses
1
1.5
2
2.5
3
3.5
4
4.5
5
POM Blood MSK CV Resp Renal Endo Integr RMEBM
2000 2001 2002 2003 2004 2005 2006 2007
Excellent
Poor
Good
V.Good
Fair
Mean Scores on Certifying Evaluations (System Courses-Yr 1; SB vs CP)
60
65
70
75
80
85
POM Blood MSK Card Resp Renal Endo
Mea
n Sc
ore
SB (93-96)CP (97-07)
A Jones, Associate Dean – University of Calgary
Mean Scores on Certifying Evaluations (System Courses-Yr 2; SB vs. CP)
60
65
70
75
80
85
Neuro Mind Repro GI
Mea
n Sc
ore
SB (93-96)CP (97-06)
A Jones, Associate Dean – University of Calgary
ALBERTA LEARNING GRADUATE EMPLOYMENTSURVEY 2004 GRADUATES FROM 2002 MEDICINE
Usefulness of Your Education in Achieving:
• Research Skills 80%
• Working with Others 97%
• A Desire to Learn More 93%
• Learn Independently 97%
• Awareness of Ethical Issues 97%
A Jones, Associate Dean – University of Calgary
ALBERTA LEARNING GRADUATE EMPLOYMENT SURVEY 2004
GRADUATES FROM 2002MEDICINE
• Satisfaction with the quality of teaching in your program? 100%
• Satisfaction with overall qualityof your educational experience 100%
• University of Alberta 83%
A Jones, Associate Dean – University of Calgary
ALBERTA LEARNING GRADUATE EMPLOYMENT SURVEY 2004
GRADUATES FROM 2002MEDICINE
• I would recommend the same program of study to
someone else. 100%
• Satisfaction with Relevance of Courses 96%
A Jones, Associate Dean – University of Calgary
Overall Quality of Education at U of C by Faculty: % Satisfied or Very Satisfied
0
10
20
30
40
50
60
70
80
90
100
FA HU SS CC SC HA ED KN EN NU SW LA MD
Data Source: 2002 Alberta Universities/Colleges’ Graduate Employment Survey re: 2000 Grads
PREPAREDNESS FOR RESIDENCY
1
1.5
2
2.5
3
3.5
4
4.5
5
1 2 3 4 5 6
U of C 05
All Schools 05No Opinion
Strongly Disagree
Data Source: Canadian Graduate Questionnaire 2005
1: I am confident that I have acquired the clinical skills required to begin a residency program
2. I have the communication skills necessary to interact with patients and health professionals
3. I have basic skills in clinical decision making and the application of evidence based information to medical practice
4. I have the fundamental understanding of the issues in social sciences of medicine
5. I have the ethical and professional values that are expected of the profession
6. I have the fundamental understanding of the basic disease mechanisms, clinical presentations and principles of diagnosis and management for common conditions
Strongly Agree
Agree
Disagree
“I AM SATISFIED WITH THE QUALITY OF MY MEDICAL EDUCATION”
0
10
20
30
40
50
60
70
Strongly Agree Agree No Opinion Disagree StronglyDisagree
Per
cent
U of C 03
All Achools 03
U of C 04
All Schools 04
U of C 05
All Schools 05
Data Source: Canadian Graduate Questionnaire 2003, 2004 & 2005A Jones, Associate Dean – University of Calgary
MCC TOTAL SCORE: 1992-2005
440
460
480
500
520
540
560
580
92 93 94 95 96 97 98 99 0 1 2 3 4 5
Class
Mea
n To
tal S
core
U of C
Canadian
A Jones, Associate Dean – University of Calgary
Performance on national exams
Performance on LMCC – Clinical Reasoning: 1994-2002
470
480
490
500
510
520
530
540
550
560
94 95 96 97 98 99 0 1 2
Class
Mean
Sco
re
U of C
Canadian
A Jones, Associate Dean – University of Calgary
720 – U of C888 - Canadian Grads/Canadian Trained
A Jones, Associate Dean – University of Calgary
328 – U of C
111 - Canadian Grads/Canadian Trained
MCC Subscale & Final Scores– Class 2003
Canadian (84), International Students (7), Total (91)
360
380
400
420
440
460
480
500
520
540
560
C leo Med O & G P eds P hello P sy S urg MC Q C R S Final
Mea
n S
core U of C Ca n
Inte rna tiona l
U of C Tota l
M CC M e a n
A Jones, Associate Dean – University of Calgary
CaRMS: PERCENT OF MATCHED STUDENTS MATCHING TO FIRST CHOICE DISCIPLINE IN 1ST ITERATION
CLASSES 2001- 2005
50
55
60
65
70
75
80
85
90
95
100
Per
cen
t
Mem Dal McG Ott Qu TO McM West Man Sk AB Cal BC
1 2 3 4 5
Data Source: CaRMSA Jones, Associate Dean – University of Calgary
RESIDENT DIRECTORS’ ASSESSMENT OF GRADUATES (PGY1)
“Overall Performance - ability to function as a resident with a full workload”
6
32
62
4
32
64
44
56
1
37
62
40
60
0
10
20
30
40
50
60
70
80
90
100
Per
cen
t
2000 2001 2002 2003 2004
Weaker than most Similar to most residents Stronger than most
Data Source: Program Directors’ Survey
Class 2000 N = 50 (71%); Class 2001 N = 45 (68%) Class 2002 N = 40 (57%); Class 2003 N = 79(90%)Class 2004 N = 76 (82%)
A Jones, Associate Dean – University of Calgary
Resident Program Directors’ Assessmentof 2006 Graduates
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Medic
al Knowle
dge
Clinic
al Ju
dgmen
t
Patient M
anagem
ent
Clinic
al Ski
lls
Profe
ssion
al D
emea
nour
Human
istic
Qual
ities
Presen
tatio
n Ski
lls
Self-Aware
ness
Psycho
social S
ensi
tivity
Perform
ance O
vera
ll
Per
cen
t
Weaker Similar Stronger
Data Source: Program Directors Survey
A Jones, Associate Dean – University of Calgary
Resident Program Directors’ Assessmentof 2005 Graduates
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Med
ical
Knowle
dge
Clinica
l Ju
dgmen
t
Patie
nt Man
agem
ent
Clinica
l Ski
lls
Profe
ssional D
emea
nour
Human
istic
Qual
ities
Prese
ntat
ion S
kills
Self-A
waren
ess
Psych
osocial S
ensi
tivity
Perfo
rman
ce O
veral
l
Per
cen
t
Weaker Similar Stronger
Data Source: Program Directors Survey
A Jones, Associate Dean – University of Calgary
Resident Program Directors’ Assessmentof 2004 Graduates
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Med
ical
Knowle
dge
Clinica
l Ju
dgmen
t
Patie
nt Man
agem
ent
Clinica
l Ski
lls
Profe
ssional D
emea
nour
Human
istic
Qual
ities
Prese
ntat
ion S
kills
Self-A
waren
ess
Psych
osocial S
ensi
tivity
Perfo
rman
ce O
veral
l
Per
cen
t
Weaker Similar Stronger
Data Source: Program Directors Survey
A Jones, Associate Dean – University of Calgary
UNDERGRADUATE MEDICAL EDUCATION ALUMNI SURVEY
CLASSES of 1992-2002
A Jones, Associate Dean
OVERALL OPINION OF THE UNDERGRADUATE MEDICAL
EDUCATION PROGRAM ALUMNI SURVEY CLASSES 1992-2002
• 97% Satisfaction with the UME program at University of Calgary
• 90% Felt prepared or very prepared for Post Graduate Training
• 98% Would advise their child or child of a relative or friend interested in Medicine to apply to the University of Calgary
A Jones, Associate Dean – University of Calgary
OPNION OF ALUMNI 1992 – 2002CURRICULUM STRENGTHS AND WEAKNESSES
Instruction Strength Neither Strength or Weakness
Bedside Clinical Correlation 92% 8%
Clinical Instruction Overall
88% 11%
Small Group Problem Solving
86% 12%
Classroom Patient Presentation
83% 15%
Learning Experiences Appropriate to Educational Objectives
80% 18%
Lectures 63% 31%
Integration of Basic and Clinical Sciences
53% 33%
“A curriculum is like water. It has the tendency to seek the lowest level of energy it can reach, and without constant renewal, it will stagnate and become putrid. To avoid stagnation alone is justification for action.”
Acad Medicine Sept 1998
Why Curriculum renewal is Important
A Jones, Associate Dean – University of Calgary
Barriers to Medical School Curriculum Changes
• Already crowded curriculum• Inadequate funding• Faculty resistance• Professional ‘turf’ issues• Scheduling conflicts
Listed by North American Academic Deans:
Graber et al. Acad Medicine 1997
A Jones, Associate Dean – University of Calgary
Curriculum – A Planned Educational Experience
• Define the outcome measures.
• Create an evaluation system to be sure these outcomes are realized.
• Develop the pathways to get to these outcomes.
Allan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education
Goals for a Revised Curriculum
A revised curriculum has to be consistent with available information on clinical problem solving and reflect basic principles of adult learning.
A Jones, Associate Dean – University of Calgary
Medical students don’t remember or can’t use the knowledge they learned in the traditional basic science courses because the knowledge is structured into mental organizations that are not useful in the clinic
Barrows, 1985
A Jones, Associate Dean – University of Calgary
Problem Based Learning – Benefits
• Activate prior knowledge
• Learn in context of clinical problem
• Interest in learning stimulated
• Self directed learning encouraged
• Life long learning encouraged
Schmidt - Norman
A Jones, Associate Dean – University of Calgary
Problem Based Learning Concerns
• Problem solving skills are not augmented
• Significant gaps in knowledge occur
• Incorrect integration of basic sciences
• Tendency to engage in backward reasoning
Albanese; MitchellAcademic Medicine
A Jones, Associate Dean – University of Calgary
Clinical Reasoning
• Clinical Reasoning and clinical knowledge are interdependent.
• Effective problem solving requires a large store of relevant knowledge.
• Clinical expertise is linked to depth and organization of clinical knowledge.
A Jones, Associate Dean – University of Calgary
Problem Solving Skills In Medicine
Research has proven that experts in specific domains learn knowledge and problem solving skills for each problem simultaneously. That is, knowledge acquisition and clinical reasoning go hand-in-hand.
Schmidt et al 1992
Efforts to Help Students Improve Clinical Reasoning
Education must focus on the development of adequate knowledge structures. Teaching, coaching, supervising must strongly encourage and nurture actual knowledge organization of the students.
Knowledge keeps no better than fish
Alfred North Whitehead 1929
Clinical Reasoning and Small Group Cases
It is useful to select one model of clinical reasoning and base the tutorial discussion on it. The precise model is less important than its generic use as a framework to structure the flow of discussion. It later serves as a fall-back strategy in complicated clinical situations.
Structure of Medical Knowledge in Memory Categories and Prototypes
Both medical textbooks and classroom teaching abound in the limitless presentation of detailed lists of disorders. More often, both fail to provide a categorization scheme that is best suited for their retrieval in a clinical problem solving situation.
BordageMed Educ 1984
Types of Curricula
• Disciplinary
• Systems-based
• Problem-oriented
• Clinical Presentations based
Clinical Presentation Curriculum
Faculty
Identify Represented byIdentify
Clinical Presentation
Schematic Problem Solving Pathway
Develop
For the Process of
Core Competencies for Clinical Presentation
Clinical Reasoning
Enabling Basic Science
Objectives
Terminal Objectives
Together Represent
Graduation Competencies
Curriculum Committee
Plans and Monitors
Curriculum
Course Content
Teaching Methods
Learning Content
Evaluation
Guidelines for
Steps in Development and Dissemination of Clinical Presentation Objectives
1. Selection of clinical problem.
2. Classification system developed to help organize knowledge needed to solve the clinical problem.
3. Key Features; Discriminating features identified of prototypic prevalent disorders.
Steps in Development and Dissemination of Clinical Presentation Objectives
4. Objectives and problem solving schemes developed.
5. Distribution to Faculty for balanced input from teachers generalists, specialists, and biomedical scientists.
6. Endorsements of objectives.
Steps in Development and Dissemination of Clinical Presentation Objectives
7. Dissemination of objectives.
8. Encouragement of implementation of objectives in teaching, learning, clinical practice and problem solving
9. Monitor and evaluate the translation of objectives and problem solving schemes into practice.
Clinical Reasoning
Student Identifies Clinical Presentation
Broad Classification of Problem
Identify Causal Alternatives and Discriminating Key Factors
Differential Diagnosis
Diagnosis
Management Plan
Schematic Problem Solving Pathway
The Scheme
• Causal Categories- pre, post and renal causes of acute renal failure
• Diagnoses- specific diagnoses for each causal category
• Basic sciences- Integral part
- Timely presentation of content
“Ask any physician of 20 years standing how he has become proficient in his art and he will reply, by constant contact with disease; and he will add that the medicine he learned in schools was totally different from the medicine he learned at the bedside.”
Wm. Osler 1932
Bleeding Tendency/Bruising
Hx PE DDx Invest. NatHx Mgmt
General Objectives
Thrombocytopenia
DisorderedPlatelet Function
CongenitalCoagulation Disorders
Acquired CoagulationDisorders
VascularAbnormalities
W SurgeryV Student
U RadiologyT Psychiatry
S PhysiologyR Pharmacology
Q PediatricsO Pathology
N OncologyM Office of Medical Education/Informatics/Culture, Health and Illness
L Office of Medical BioethicsK Obstetrics & Gynecology
J NeuroscienceI Microbiology
H MedicineG Immunology
F GeneticsE Family Medicine
D Community Health Sciences/Nutrition/PreventionC Biochemistry
B AnesthesiaA Anatomy
Bleeding Tendency/ Bruising
History Physical Examination
Differential Diagnosis
Investigation
Natural History, Prognosis &
Complications of Condition
Prevention, Treatment &
Complications of Treatment
GeneralObjectives
Thrombocytopenia
Disordered Platelet Function
Congenital Coagulation Disorders
Acquired Coagulation Disorders
Vascular Abnormalities
a
b
c
d
e
f
g
0 1 2 3 4 5 6
Schematic Problem Solving Pathway
Bleeding Tendency/ Bruising
Platelets Coagulation Vascular
Decreased Number
Abnormal Function
Congenital Acquired Congenital Acquired
Clinical Presentations
Broad Classification of Problem
Causal Alternatives and Discriminating
Key Factors
Differential Diagnosis
Basic Science Objectives for Bruising and
Bleeding
Pathology (Vessels)
Anatomy (Spleen)
Histology (Bone Marrow)
Pharmacology (ASA, Heparin)
Immunology (ITP, Vasculitis)
Genetics (Hemophilia)
Physiology (Hemophilia)
Basic Science or Biomedical Knowledge in the Undergraduate
Program The purpose of basic science teaching is to
provide a scientific foundation for tasks of clinical practice such as diagnosis and therapeutics. The essential challenge of balancing depth of understanding with breadth of coverage remains.
(See p. 35, Fig. 4.1)
W SurgeryV Student
U RadiologyT Psychiatry
S PhysiologyR Pharmacology
Q PediatricsO Pathology
N OncologyM Office of Medical Education/Informatics/Culture, Health and Illness
L Office of Medical BioethicsK Obstetrics & Gynecology
J NeuroscienceI Microbiology
H MedicineG Immunology
F GeneticsE Family Medicine
D Community Health Sciences/Nutrition/PreventionC Biochemistry
B AnesthesiaA Anatomy
Bleeding Tendency/ Bruising
History Physical Examination
Differential Diagnosis
Investigation
Natural History, Prognosis &
Complications of Condition
Prevention, Treatment &
Complications of Treatment
GeneralObjectives
Thrombocytopenia
Disordered Platelet Function
Congenital Coagulation Disorders
Acquired Coagulation Disorders
Vascular Abnormalities
a
b
c
d
e
f
g
0 1 2 3 4 5 6
Fever
Fever
< 2 weeks > 2 weeks
Infectious
Bacterial Viral
ACUTE VISUAL LOSS
PreRetinal• Corneal edema (glaucoma)• Vitreous hemorrhage (Diabetes)
Retinal• Acute Macular Lesion (hemorrhage)• Retinal Detachment (spontaneous)• Retinal Artery Occlusion (carotid emboli)
Post Retinal • Optic Neuritis (MS)• Ischemic Optic Neuropathy (Temp. Arteritis)• Occipital Infarction/Hemorrhage
Chrichton, Verstraton, Fletcher
Fluctuating Altered CognitionFluctuating Altered Cognition Delirium and Confusional States
In the Head Out of the Head
•CNS Infections•Seizures
•Hypertensive Enceph•Psychiatric Disorders
•Toxins•Metabolic Derangements•Systemic Organ Failure
•Physical Disorders
Strengths of Year I-II Curriculum
• Approaches to problem solving – clinical reasoning
• Early clinical exposure• Small group teaching• Clinical correlation; patient presentations• Basic science integration with problem
solving• IST • Medicine 440; elective time• Communication; physical examinationAllan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education
Weakness of Year I-II Curriculum
• Exams not always reflective of teaching ‘emphasis’
• Lack of pharmacology• Faculty not promoting core documents;
teaching to objectives not always clear• Small group teaching variable• Problem solving with schemes course
dependent
Allan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education
A Jones, Associate Dean
The Curriculum First Year Courses
Aug Oct Nov Dec 19 Jan.2 Mar. 26 Mar. 29 June -July 18
P FOR M-Blood-GI(Course I)
MSK-Derm-Opth-ENT
(Course II)
HOLIDAY
2 wks
CV-RESP
(Course III)
Renal-Endocrine(includes
Obesity P for M)
(Course IV)
HOLIDAY
2wks
ELECTIVE
2wks
REWRITES
MEDICAL SKILLS PROGRAM
RMEBM-Health Promotions-Disease Prevention-Population Health
Number of weeks for First Year = 45 weeks
A Jones, Associate Dean
The Curriculum
Second Year Courses
July 18
Aug. 18 Oct. 10 XMAS Jan. 3Feb. 6 Mar. 5
ELECTIVE
4wks
Neuroscience-Aging
(Course V)
Infant-Child-Reproduction-
Genetics(Course VI)
HOLIDAY
2 wks
Mind-Family(Course
VII)
IntegrativeCourse
Introduction to Clerkship
Teaching Methods
• Lectures for rapid acquisition of key content
• Small group case based learning- In depth self-directed (or guided) learning
- Review, reinforcement, practice and feedback
- Problem solving, motivation, pertinence
• Clinical correlation- Bedside sessions
Clinical Reasoning
Clinical reasoning does not develop in isolation: it is associated with increasingly refined and elaborated medical knowledge. Problem solving is domain-specific and not generic, so the challenge for medical educators is not only to make explicit the process of reasoning but also to identify the necessary content.
Schmidt et al 1990
Kassirer 1995
Usual Sequence of Instruction
• Presentation and scheme shown- Case based or otherwise
• Lectures or PBL sessions planned- In depth knowledge and acquisition
- Basic and clinical sciences
• Small group sessions for reinforcement- Shorter case scenarios for review
- Clinical correlation
Advantages of New Curricular Structure
• Courses will be linked to graduation objectives and UME program philosophy of teaching, learning and evaluation.
• Linkage of courses will integrate CP better and reduce redundancies.
Curriculum
Task Force Report
Advantages of New Curricular Structure
• Clinical presentation list will be revisited and clerkships will adopt appropriate presentations.
Curriculum
Task Force Report
Scheme Use Reported by First & Second Year Students(Classes of 2007 and 2006)
1
1.5
2
2.5
3
3.5
4
4.5
5
Learning Prob-Solving Often Referred Integration
1st Yr 2nd Yr
Strongly Agree
Strongly Disagree
Neutral
Data Source: Classes 07 & 06 yr end CP curriculum evaluation
MAP OF DEPARTMENT INVOLVEMENT IN UME CURRICULUM*COURSES IN YEARS I AND II
Department I II III IV HPOP V VI VII Int. Intro Clerk
MedSkills
Medicine X X X X X X X X X
Family Med X X X X X
Pediatrics X X X X ? X
Psychiatry X X X X
Surgery X X X X X ?
Neurosci X X
Anatomy X X X X X
Oncology X X X X
Obst-Gyn X
Community Health
X
Medical Genetics
X
Pathology X X X
ER X X X
Anesthesia X
NUMBER OF HOURS TAUGHT IN THE UNDERGRADUATE MEDICAL PROGRAM YEARS I-II
GFT and CLINICAL FACULTY
Hours Relative Dept Size Average Teaching hrs per Member
Medicine 1844 7
Family Med 1189 7
Pediatrics 694 5
Neuroscience 514 8
Psychiatry 458 4
Surgery 437 3
Pathology 257 4
Anatomy 252 10*
Oncology 242 3
Obst-Gyn 155 6
Com Health 78 1
Med Genetics 51 4
LEARNING METHODS USED DURING YEARS I AND II UNIVERSITY OF CALGARY
UNDERGRADUATE PROGRAM
Small Groups/Bedside: 1/3
Large Groups/Classroom: 1/3
Self Directed Study: 1/3
A Jones, Associate Dean
SUMMARY OF STRUCTURED TEACHING HOURSUNDERGRADUATE MEDICAL EDUCATION PROGRAM
YEARS I AND II 2004-2005
Hours %
Non-GFT 5316 68
GFT 2453 32
TOTAL 7769 100
A Jones, Associate Dean
*Medicine 440, Summer Elective not included
Ownership of the Curriculum
"Faculty members own what is taught in the curriculum - they own the content. The Associate Dean and Curriculum Committee are responsible for the methods and the effectiveness. Faculty give the course, the Associate Dean has to give the degree.”
Academic MedicineVol. 73, Pg. 54
The Seven Most Dangerous Words in Medical Education
“…but we’ve always done it this way”
Allan R Jones, MD FRCPC, Associate Dean, Undergraduate Medical Education
1. Decide what to do.
2. Build support for the idea.
3. Acquire resources.
4. Do it wrong the first time.
5. Do it wrong the second time.
6. Do it passably the third time.
7. Do it reasonably well the fourth time.
Friedman Acad Medicine 1993
Prototype Timeline for a Major Educational Change