Overview of Newer Educational Techniques and Evaluation Options · 2012-04-13 · Overview of Newer...

Overview of Newer Educational Techniques and Evaluation Options

Judy A. Shea, PhD.Associate Dean of Medical Education Research

Professor of Medicine, Clinician-Educator

To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.

Overview

Mission of medical education produce clinicallyMission of medical education – produce clinically competent physicians

Many players:y p y» Learners» Educators » Administrators» Evaluators

V l i il d i t Value similar endpoints» Have different foci, priorities, agendas


Objectives T i d ti l t h i To review some newer educational techniques To present a framework for thinking about medical

education research/evaluationeducation research/evaluation To review evaluation options/issues To talk a little bit about program evaluation


Educational TechniquesEducational Techniques

Learner EngagementLearner Engagement“How do we teach?”

C t tContent“What do we teach?”


Learner Engagement“How do we teach?”

“How of teaching” How of teaching» Active learning» Practice» Spaced learning

Simulation – SPs Essays, writing, reflection Self-assessment Technology – fewer books Less lecture, more hands on


Content“What do we teach?”

Data access retrie al interpretation Data access, retrieval, interpretation Social and economic sciences I t l kill d i Interpersonal skills and issues “Special topics”

EBM» EBM» Patient safety» Geriatrics women’s health» Geriatrics, women s health

Business of medicineTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.

What You (Really) Need to KnowLawrence Summers – NYT – 1/20/12

More abo t ho to process and seMore about how to process and use information and less about imparting it

CollaborationCollaborationNew technologies change how knowledge is

conveyedconveyed“Active learning classrooms”English’s emergence as a global languageEnglish s emergence as a global languageMore emphasis on analysis of data


Medical Education Research/Evaluation Framework

What don’t I mean?What don t I mean?NOT a conceptual model, e.g.,

Adult learning theorySocial learning theoryBehavioral (economic) theoryTheory of Planned BehaviorTheory of Planned BehaviorTheory of Reasoned Action

Much simpler…pwhat and who are you studying?


Medical EducationMedical Education Research/Evaluation Framework

WHOWHAT

Focus of ResearchUnit of Analyses Process Outcome

P ( dPerson (student, resident, faculty)

x x

Program (course, residency)

x x


Focus of Research/Evaluation

Unit of Analyses Process Outcome

P ( t d t T h t d j i A t i h hPerson (student, resident, faculty)

To what do junior faculty/fellows attribute their success and how does mentoring fit in?

Are trainees who have mentors more

successful than peers who do not?g

Program (course, residency)

How can mentoring activities provided in IM

training programs be

Do programs with structured mentoring programs produce

characterized?.

more academic clinicians than those

without?To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.

Reason for the Framework Clarity Focus


Evaluation QuestionsEvaluation Questions

WHY?WHY? » What are you trying to do?

HOW?HOW? » What evidence would be convincing?


Miller’s Pyramid


WHY? - Evidence - K,A,S (B) – Typical/Old

Kno ledgeKnowledge» Usually ‘written test’

AttitudeAttitude» Often survey

Skills/BehaviorsSkills/Behaviors» Self-report


WHY? Evidence - K,A,S (B) – CurrentKnowledgeKnowledge

» Usually ‘written test’, chart-stimulated recallAttitudeAttitude

» Often survey, observations, peers and patient ratings

S ( )Skills (a.k.a. competencies)» Simulations, technology (e.g., actiwatch)

BehaviorsBehaviors» Claims data, chart review, GPS, time-motion


HOW? Evaluations Tools/DevicesHOW? Evaluations Tools/Devices

Sim lation SPs OSCEs Simulation – SPs, OSCEs Essays, writing, reflection S lf t Self-assessment Technology – actiwatch, GPS Portfolios Observation – mini-CEX, DOP, 360o


HOW? Primary Data Qualitative Qualitative

» Focus groups» Interviews» Interviews» Observational data

Quantitative» Surveys» ‘tests’

Chart reviews


HOW? Secondary Data

Secondary data » Grades, ratings, career choice» Tests – MCAT, NBME, ABIM» AMA database

W b b d l ti» Web-based evaluations


Ad d Di d fAdvantages and Disadvantages of Typical Methods

Knowledge PE skills

MCQ test ++ -

SP based test -- ++

Teachers/attendings + +

Chart reviews ? ?


Advantages and Limitations of Four “”Newer” Methods

Obser ed enco ntersObserved encounters360° degree assessments P ti t/t i tPatient/trainee outcomesProcess of care/teaching


Advantages of Encounter-based Methods

Pose a broad range of patient/trainee problems Pose a broad range of patient/trainee problems Support the assessment of integrated skills Support education Support education Feasible in small work-based practices and training

programs


Limitations of Encounter-based Methods

Not many encounters will be considered Not many encounters will be considered unsatisfactory

If the trainees-doctors have some control over who examines them and indirectly over the content of the assessment

Standards across practices and programs will not be Standards across practices and programs will not be equivalent

A large scale faculty development effort is neededg y p


Advantages of 360° AssessmentAdvantages of 360 Assessment

S pported b considerable researchSupported by considerable researchHas positive educational effects

S t lit i t t l» Supports quality improvement tools FeasibleA li bl th tiApplicable across the continuum


Limitations of 360° AssessmentLimitations of 360 Assessment

Assessor Assessor» Need to have observed the doctor» Differ in stringency and perspective

Context» Clinical settings differ in demands, stresses, etc.

Relationships Relationships» Competition, friendship may influence assessments

Stakes» In high stakes settings, assessments might be inflated

Anonymity is importantTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.

Advantages of OutcomesAdvantages of Outcomes

P blicPublic» Measure of accountability

Patients trainees health care systemPatients, trainees, health care system» Offers a basis for deciding among

doctors/teachers on quality/efficiencyq y yDoctors/teachers

» Offers assessment tailored to unique practice q p» Based on real work performance


Limitations of OutcomesLimitations of Outcomes

AttributionAttribution» Good assessment requires that the doctor/teacher

be solely responsible for outcomesCase/content mix and patient/trainee

complexity» Good assessment requires that all face the same

challengeNumbersNumbers

» Reliable assessment requires that many patients/trainees be sampled


p p

Advantages of Process MethodsAdvantages of Process Methods

More directl in doctor/teacher controlMore directly in doctor/teacher control » Problems of attribution are reduced

Less directly influenced by complexityLess directly influenced by complexity» HbA1c should be monitored in all diabetics» Time spent teaching» Time spent teaching

For certain processes, case mix is not a problemproblem» Most patients need immunizations


Limitations of Process Methods

Doing the right things does not ensure a good outcome Doing the right things does not ensure a good outcome Complexity and case mix still have an effect A sizeable number of a teacher/doctor’s A sizeable number of a teacher/doctor s

patients/trainees need to be sampled


Two more issuesTwo more issues..

DesignsDesignsAmount of data


Designs Correlational/cohort studies Correlational/cohort studies

» observational» descriptive» descriptive

Experimental studies» intervention» post-intervention observation


Non-experimental Si l t t t l Single group - post test onlyG1: -------------I-------------O Single group – pre-post test Single group – pre-post testG1: O-----------I-------------O

Extremely common in medical educationMultiples threats to validity


Quasi experimental

Have a nonrandom control groupG1: UC OG1:________UC________OG2:________I__________O

Probably underutilized in medical education


Experimental Subjects are randomly assigned to treatment groups Some level(s) of blinding

RG1:_______UC________ORG2: I ORG2:_______I__________O


Amount of Data

Need adequate number of observations Case specificity Case specificity Sampling/bias Capture more signal than noise Capture more signal than noise


Where is the sampling error?

MCQ test Wrong content, not enough items

SP based test SP training/specificity

Teachers/attendings Different ideas/ “hawks” and “doves”

Chart reviews A lot does not get recordedChart reviews A lot does not get recorded


***Need a lot of observations***

Purposes of Program EvaluationPurposes of Program Evaluation

To impro e programTo improve programTo determine next steps/make decisions

H l d id t l d l f th li i t» Help decide to replace, develop further, eliminate, accredit

To determine effectivenessTo determine effectivenessTo document successTo measure outcomesTo measure outcomes


Many ModelsMany Models

Logic Model Goal Oriented/Objective-Based (Tyler) Kirkpatrick’s 4-level Framework Continuous Quality Improvement (CQI) and Plan-Do-

Study-Act (PDSA) - Goals-free Evaluation (Scriven)

/ Judicial/Adversary Evaluation CIPP (Stufflebeam) Situated Evaluation Connoisseurship Evaluation (Eisner) Utilization-Oriented Evaluation (Patton)


Program Logic Model - MERCProgram Logic Model - MERC


Tyler Model - MERCTyler Model - MERC

Objectives: Methods Content/Specifics Frequency/ PersonTiming

1. Increase their participation in medical education research

Short survey

Retrospective pre/post survey•12 closed-ended dichotomous items participation in medical d i h i i i

6-12 months after completion of MERC

MERC Evaluation Committee to launch and analyze data

activities (research presentations and publications)(outcome evaluation)

education research activities (ie, collaborating in medical education research project, publishing a peer-reviewed publication)

2. Apply medical Short

Open-ended question

6-12 months after completion MERC Evaluation pp y

education research principles from MERC to their daily work(outcome evaluation)

surveyp

of MERC Committee to launch and analyze data


Plan-Do-Study-Act CyclePlan-Do-Study-Act Cycle


Overview of 10 Program Evaluation Steps (Workplan)

Step 1: Identify Users Step 1: Identify Users Step 2: Identify Uses Step 3: Identify Resources Step 4: Identify Evaluation Questions/Objectives Step 4: Identify Evaluation Questions/Objectives Step 5: Choose Evaluation Design Step 6: Choose Measurement Methods and

Construct InstrumentsConstruct Instruments Step 7: Address Ethical Concerns Step 8: Collect Data Step 9: Analyze Data Step 9: Analyze Data Step 10: Report Results


Step 1: Identify UsersStep 1: Identify Users

Who will use the evaluation?Who will use the evaluation?» Learners» Faculty» Faculty» Workshop developers» Administrators» Agencies» Other stakeholders

What do they want from the evaluation?


Step 2: Identify UsesStep 2: Identify Uses

Generally both formative and summativeGenerally both formative and summativeIndividual and program decisionsConsider specific needs of each userConsider specific needs of each user

» Judgments about individuals» Judgments about project management and» Judgments about project management and

processesImprovement or proving?p p g


Step 3: Identify ResourcesStep 3: Identify Resources

What time is needed from e er one?What time is needed from everyone?

What personnel is needed?p

What equipment?

What facilities?

What funds?What funds?


Step 4: Identify Evaluation Questions/Objectives

These go back to the model chosen butThese go back to the model chosen, butGenerally

» Relate to specific measurable objectives forp j• Learner• Process• Outcomes

» Wise to include some questions that get at what was not anticipated both as strengths and weaknesses


Step 4: Identify Evaluation Questions/Objectives – cont.

Evaluation questions should:» Be clear and specific» Congruent with the literature» Focus on outcomes versus process

Outcomes imply change» Outcomes imply change• Workshop will improve educator’s skill

RATHER THAN• How the workshop will be given (process)

» Align with goals and objectivesTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.

What are the questions? Interface•Interaction

Process•Ease of use Presentation &

Interaction•Feedback•Clarity•Quality•Organization

Evaluation of Learning

Outcomes

Ease of use•Efficiency•Relevance•Language

Organization

Pedagogy•Instructional method

Organization

?Outcomes

•Knowledge•Attitudes•Behaviors

E l ti

•Instructional method•Structure•Active learning•Learner differences•Objectives ~ methods

Evaluation of Content

Evaluation of Cost

D l t I l t ti M i t

Objectives methods

•Authority•Accuracy•Appropriateness•BreadthD h

Development Implementation Maintenance•Needs assmt•Objectives•Materials

•Staff time•Materials•Recruitment

•Portability•Coordination•Durability

To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content. •DepthMaterials•Staffing•Design

Recruitment•Facilities•Hardware

Durability•Tech support

Adapted from Elissavet & Economides (2003)

Step 5: Choose Evaluation DesignsStep 5: Choose Evaluation Designs

What ones are appropriate to the questions?What ones are appropriate to the questions?

» Posttest only X - - Oy• Satisfaction/reactions

P t t P tt t» Pretest-Posttest O - -X- - O• Changes in knowledge/attitudes

» Quasi-Experimental O - -X- - O- - - - - O • Cross-over O - - - - - O - -X - - O


Step 6: Choose Measurement Methods pand Construct/Adapt InstrumentsCommon methods

» Rating forms» Self-assessments» Essays» Exams» Questionnaires» Interviews/focus groups» Direct observations» Performance audits» Existing data (AAMC ti i C l JAMA)» Existing data (AAMC questionnaires, Course evals,JAMA)

Collect appropriate demographicsTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.

Step 7: Address Ethical ConcernsStep 7: Address Ethical Concerns

ConfidentialityAccess to dataConsent Resource allocationSeek IRB approval


Step 8: Collect DataStep 8: Collect Data

Timing and response rate

Already existing data collectionAlready existing data collection

Impact on instrument design (e.g. mail vs. web survey)web survey)

Assignment of responsibility


Step 9: Analyze DataStep 9: Analyze Data

Plan at the same time as the rest of thePlan at the same time as the rest of the evaluation

Want congruence between question asked and analysis that is feasible


Step 10: Report ResultsStep 10: Report Results

TimelTimely

Format fits needs of users

Display results in succinct and clear manner


Other Issues

How many data collection points Sustained change Fairness in ‘taking credit’ Teams


Thi I Did N t T lk Ab tThings I Did Not Talk About

U i h ll i d ti Unique challenges in education:» real deadlines and timelines» small samples» small samples» many “competing” curricula» teams» apprenticeship learning

Abundant opportunity and need to conduct medical education evaluation research


Overview of Newer Educational Techniques and Evaluation Options · 2012-04-13 · Overview of Newer...

Documents

Transcript of Overview of Newer Educational Techniques and Evaluation Options · 2012-04-13 · Overview of Newer...