Resident Evaluation of a Web-based Integrative Medicine Curriculum: The Function of Formative Evaluation

Ben Kligler, M.D., Patricia Lebensohn, M.D., Sally Dodds, Ph.D., Raymond Teets, M.D., & Victoria Maizes, M.D.

Presentation Objectives

The objectives of this presentation are to:

▫ Describe the function of formative evaluation for curriculum development.

▫ Review the results from learners’ evaluation of the Integrative Medicine in Residency curriculum.

▫ Describe the procedures used in applying feedback from learners’ evaluations to guide curriculum changes and revisions.

Integrative Medicine in Residency (IMR) is…

• Competency-based, online, 200-hour, curriculum.• In-depth training in Integrative Medicine.• Incorporated through all 3 years of Family Medicine

residency.• Piloted at 8 residencies nationwide.• Seamless, online evaluation of the curriculum and the

residents.• Responds to ACGME competency requirements.• Evaluation developed simultaneously with the

curriculum.

University of Arizona

University of Texas Medical Branch

Hennepin County

Carolinas Medical Center

Beth Israel

Maine-Dartmouth

Maine Medical Center

University of Connecticut

IMR Program Locations

Tools in Integrative Medicine

Prevention and

Wellness

Women’s Health

Chronic Illness

Pediatrics

Acute Care

IMR Curriculum Units

Introduction to Integrative Medicine

Prevention and Wellness:• U.S. Preventive Health Services• Nutrition and Diet • Supplements for Prevention• Physical Activity• Sleep • Stress and Mind-Body Medicine • Spirituality• Clinical Integration

Tools in Integrative Medicine:• Integrative Medicine Intake and Care Plan • Botanicals• Mind-Body Medicine• Manual Medicine • Introduction to Energy Medicine and Whole

Systems • Practice Management• Motivational Interviewing for Behavioral

Change

Acute Care:• Acute Back Pain,• Urinary Tract Infection,• Gastroenteritis,• Otitis Media,• Vaginitis,• Atypical Chest Pain,• Upper Respiratory Infection

Pediatric Topics:• ADD/ADHD• Chronic Pain Syndrome• Asthma and Allergies

Women’s Health Topics:• PMS/PMDD• Dysmenorrhea• Menopause• Fibromyalgia• Osteoporosis • Depression • Eating Disorders • Pregnancy and Lactation

Chronic Illness: • Cardiovascular Disease• Type II Diabetes• Osteoarthritis • Rheumatoid Arthritis• Obesity • Irritable Bowel Syndrome• Chronic Back Pain

Special Topics:• HIV/AIDS • Cancer Survivorship• Environmental Medicine

Units and Courses

IMR: Educational Methods Needs assessment informed curriculum design. Web-based curriculum written and edited by

Integrative Medicine educators. Competencies aligned with the ACGME Outcomes

Project. Flexible modular format to meet the needs of

residency schedules. Case-based, interactive learning and streaming

video. Experiential exercises and process-oriented group

activities at the residency sites. A community of learners through online dialogues

with faculty and peers. Emphasis on teaching and promoting physician

well-being and self-care.

IMR User Interface

Online portfolios

Flexible modular format

Resources and Links

Direct Observation Checklists

Reflections

“When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative.” - Robert Stakes

What is Formative Evaluation?

Formative Evaluation…

• Helps form and strengthen programs by:▫ Examining the need for them,▫ Their delivery or technology▫ The quality of their implementation▫ Assessing their organizational contexts, procedures, and

resources (Scriven, 1991).

• Methods emphasize data collection and analysis prior to completion.

Why Use Formative Evaluation in the IMR?

• Medical knowledge in Integrative Medicine is dynamic, evolving rapidly; revisions are largely driven by new information.

• Distributed, web-based curricula requires feedback from learners to be successful.

• In the IMR, formative methods include:1.Preliminary needs assessment2.Feedback from resident evaluation of courses

1. IMR Needs Assessment(Benn, Maizes, Guerrera, Sierpina, Cook, & Lebensohn, 2009)

• Methods▫ 222 faculty and residents from 8 family medicine

programs (60.2% response rate).▫ Online survey w/ structured and open-ended questions.

• Results▫ Preferred IM be woven throughout all curriculum areas

(67%).▫ Top topics: Nutrition, Supplements, and Physician

Wellness.▫ Top curricular areas for IM enhancement: Chronic

illness, behavioral health, and outpatient medicine▫ Viewed IM central to family medicine training, patient

care, and the field of family medicine (84%). ▫ Top challenges: Limitations in time, resources, and

acceptance.

2. IMR Course Evaluations

• Measures assess learners’ ratings of the course in:▫ Meeting course objectives▫ Clinical utility of the course▫ Time needed to complete the course▫ Functionality of the online technology

Analyzed when 50% of pilot residents complete a course.▫ Ratings < 8o% and open-ended comments targeted for review.

Review of 01 pilot group suggested Likert-type response categories be changed from 4 to 5 points to increase precision and variability.

Resident Demographics2011 (n =

67)2012 (n = 64) Controls (n = 31)

SexFemaleMale

64%36%

64%36%

45%55%

Mean Agerange

32 (27 – 43)

32(24 – 56)

30(26 – 38)

Marital StatusMarriedLive w/ partnerSingleUnk.

45%13%27%15%

27% 6%34%33%

58%10%32%0%

Graduate StatusUS MDUS DOFMGUnk.

51%18%27% 4%

41%26%28% 5%

55%16%16%13%

Did Course Meet Learning Objectives?Year 01 Content

Intro

Prev

en...

Nut

rition

Suppl

em...

Phys

Ac...

Stres

s...

Sleep

Spirit

ua...

Clin In

teg

92%94% 94%

82%

91%

95% 94%

89%

95%92%

94%96%

91%

Very definitely/Definitely

2011 2012

*Currently, data are available for 4 courses for the 2012 class.

Clinical Utility of the Course?Year 01 Content

Intro

Prev

entio

n

Nut

rition

Suppl

em...

Phys

Ac...

Stres

s...

Sleep

Spirit

ualit

y

Clin In

teg

78%

95% 93%99% 96%

89% 91%

47%

93%

74%

96% 97%86%

Very Useful/Useful

2011 2012

*Currently, indicators are available for 4 courses for the 2012 class.

How Smooth was the Online Technology?Year 01 Content

Intr

o

Preve

n...

Nutr

ition

Supplem

...

Phys A

c...

Stres

s...

Sleep

Spiritu

a...

Clin In

teg

78%

91% 95% 97% 94% 90% 93%

81% 85%82% 83%94% 94%

Very Smoothly/Smoothly

2011 2012

*Currently, indicators are available for 4 courses for the 2012 class.

Meet Learning Objectives? Clinical Utility?

ADHD Allergy, Asth

Pedi Pain

PMD/PMDD

100% 98% 100%94%

Very Useful/Useful

ADHD Allergy, Asth

Pedi Pain

PMD/PMDD

97% 98% 100% 100%Very Definitely/Definitely

Year 02 Course EvaluationsCourses currently at 50% threshold

Year 02 Course EvaluationsCourses currently at 50% threshold

Smooth Online technology?

ADHD Allergy, Asth

Pedi Pain

PMD/PMDD

92%

82%

100%

77%

Very smooth/Smooth

Length of Time SpentYear 01 &Year 02 Content

Class Course < 1 hr 1-2 hr 2-3 hr 3-4 hr 4-5 hr 5-6 hr

2011 Introduction 34% 44% 13% 5% 4%

2012 34% 36% 26% 4%

2011 Supplements 32% 35% 12% 12% 5% 3%

2012 50% 36% 9% 5%

2011 Nutrition & Diet 13% 25% 31% 16% 6% 7%

2012 21% 42% 29% 4% 4%

2011 Physical Activity 61% 38% 2%

2012 66% 28% 7%

2011 Stress & Mind Body 34% 43% 17% 3% 2%

2011 Spirituality 72% 26% 2%

2011 Sleep & Health 94% 6%

2011 Preventive Services 59% 32% 7%

2011 Clinical Integration 17% 38% 21% 8% 9% 8%

2011 Allergy/Asthma 11% 43% 27% 11% 3% 5%

2011 ADHD 48% 48% 4%

2011 Pediatric Pain 55% 27% 14% 4%

2011 PMD/PMDD 53% 40% 7%

Year 1

Year2

What Residents Say…

I enjoyed the interactive nature of the modules.

The tools are amazing!Did an excellent job of relating

spirituality to health care.I think sleep is one thing most

neglected in health care topics. This course has rightly identified it.

The ability to self reflect as well as think about our patients.

I loved the video lecture – very engaging and interesting.

This is a very comprehensive topic.Great links and resources.The seven cases. Great way to pull all

of these courses together!

The buttons aren’t working.There was a lot of data in this module

[Nutrition and Diet], but not enough interactive learning.

Too lengthy and too in-depth.Some links were not available.Too many resources were given and I

did not have enough time to thoroughly review them.

Too much reading, too many links, and too many videos.

Discussion

AcknowledgementsUniversity of Arizona Pilot Sites

Emily Sherbrooke Victor Sierpina, M.D.

Paula Cook Selma Sroka, M.D.

Rhonda Hallquist Mary Guerrera, M.D.

Tieraona Low Dog, M.D. Dael Waxman, M.D.

Craig Schneider, M.D.

John Woytowicz, M.D.