Educational Outcomes Evaluation:

The Future for CME

James C. Leist, EdD

Staff Consultant

Alliance for Continuing Medical Education

Disclosure

Part time CME consultant

This format and some of the content were developed by Donald Moore, Vanderbilt University, Robert Addleton, Medical Association of Georgia and Derek Dietze, Improve CME

Plan for Session

Challenge-based Approach– Scenario– Independent Reflection– Small and Large Group Work– Mini Lecture– Further Reflection and Integration

Challenge Scenario

In her recent meeting with the Hospital CEO, Maureen O. Measure, BSN, MEd, the CME Director for Hopewell General Hospital, was asked about the value of CME. Maureen noted that they have done many programs for all physicians in the hospital system.

The CEO asked whether CME improved physician practice and patient care.

Maureen said that she didn’t have any data.

Challenge Scenario

The CEO asked if QI data were needed and agreed to appoint her to the QI Committee for the Hospital. She agreed anxiously.

In her third meeting with the QI Committee, Maureen listened to a report about the varied levels of physician prescriptions of ACE inhibitors for patients with CHF. How could the physician behavior be changed to assure that at least 80% of patients received the appropriate dosage of ACE I?

Challenge Scenario Questions

What can a CME provider contribute to this scenario?

If a CME intervention is provided, what outcomes should be evaluated and how?

ChallengeIndividual Reflection

For the next 5 minutes, reflect on the challenges listed below:– What are Outcomes?– How do you evaluate the outcomes of a

CME activity for this scenario?

ChallengeSmall Group

In your small groups in the next 10 minutes, share the results of your individual reflection on the definition of outcomes and how you would address the outcomes of a CME activity for this scenario.

Designate one member of the small group to share your results with the large group, time permitting.

Outcomes Evaluation Mini-Lecture

Learning Objectives for session– Define Educational Outcomes– Describe how CME providers in the

Hospital/Health Systems Venue can contribute to more sophisticated educational outcomes

– Apply measurement of educational outcomes to one CME activity when you return to your venue

Outcomes Defined

An outcome is defined as– the result, or effect, of an event– the consequences of an action

In CME, an educational outcome is the result, or effect, of CME activities

Evaluation is the process for measuring outcomes

Key to Outcomes Evaluation is to plan for the results that are desired – NEEDS ASSESSMENT DATA ARE CRITICAL

Yogi Berraism

“If you don’t know where you’re goin’, you ain’t gonna get there”

“Yogi Berra”

Process for Evaluating Outcomes

Conduct a needs assessment - practice gap Determine if education can address the gap Identify the results of the educational intervention Develop/Implement the intervention/s - target

audience and Faculty Measure the results/outcomes Share the results with appropriate stakeholders Repeat and Improve the process

Practice Gap - Individual and/or Organizational

Current Practice(Problem/Need) - Current level Desired Result - Practice/Performance level

desired Gap - The Difference between the desired result

and the current practice Will an educational intervention/interventions

address the identified gap.

A Continuum, or Levels, of Educational Outcomes/Results

Attendance Satisfaction Learning - KSA (Knows) Competence (Knows/Shows How) Performance/Behavior in Practice -

Individual and Organizational (Does) Patient Outcomes Community Health Status

Evaluation - A Continuum of Educational Outcomes

Current Practice - Generally– Attendance– Satisfaction– Learning (Knowledge, Skills and Attitudes)

Most Desirable, More Difficult and Required– Competence– Performance/Behavior in Practice - Individual and

Organizational– Patient Outcomes

Outcome Evaluation by Venues

Venues

Results

Office-based

Practice

Hospital/

Health System

Specialty

Societies

Med. Ed.

Comm. Co.

Medical School

Attendance/

SatisfactionX X X X X

Learning - KSA

X X X X X

Competence X X X X X

Practice X X X? X? X?

Patient Outcomes

X X X? X?

Community X X X X?

How do we Evaluate Outcomes?

Accurate data-based needs assessment of desired/current practice required to document need for change and provide baseline for measurement of improvement in practice.

Educational Formats Data Collection Strategies

– Indirect – Self-report by physicians– Direct – Actual performance data that describes

performance, or result of the performance

Educational Formats

Conferences– Advantages

– Disadvantages

Regularly Scheduled Series - RSS

– Advantages

– Disadvantages

Data Collection Strategies - Indirect

Outcome – Physicians will prescribe ACE inhibitors in the appropriate dosage for patients with CHF in every case when it is indicated

Measure – Prescriptions for ACE inhibitors

Methods – Survey; focus group; interview cardiologists

Data Collection Strategies - Direct

Outcome – Physicians will prescribe ACE inhibitors in the appropriate dosage for patients with CHF in every case when it is indicated

Measure – Prescriptions for ACE inhibitors

Methods – Chart audit; chart recall; prescription pads; observation; QI/PI data

Learning (Knows)

Methods for Measuring Knowledge, Skills and Attitudes– Post program evaluations

Listing of what you learned (most important thing that you learned)

– Pre- and Post - Tests– Others?

Competence (Knows/Shows How)Minimum evaluation standard

Methods for Measuring Competence– Post Test with Case Study/Scenario-Asks how– Commitment to Change with Explanation - What

do you plan to change and how?– Demonstration of how to apply a skill learned in

the CME activity– Action Plan for use of knowledge learned.– Others?

Performance/Behavior in Practice

Methods for Measuring Performance in Practice– Indirect (1-3 Months)

Follow up Survey - Self Report Observed practice Others?

– Direct (3-12 Months) Monitor Clinical Data - organizational Chart Audit - individual and organizational Others?

Patient Outcome

Methods for measuring patient outcomes– Monitor health status data for target audience– Monitor patient practices related to need addressed– Others?

Value of CME Formats(Conference/RSS) in Each Outcome Level

Learning

Competence

Performance

Patient Outcome

Challenge Scenario Questions

What can a CME provider contribute to this scenario?

If a CME intervention is provided, what outcomes should be evaluated and how?

How Do We Evaluate CME in Changing Physician Practice/Performance?

Conduct effective Needs Assessment to establish a baseline of performance, or health status, and identify the gap between what is and what should be. (EXAMPLE)

Numerous RCTs have shown that in patients with CHF, ACE Inhibitors

– Reduce mortality by 20-25%– Increase functional status and quality of life– Decrease hospitalization and death by 35-40%

ACE Inhibitors are cost effective Currently, ACE-I are prescribed for only 50% of pts.

How Do We Evaluate CME in Changing Physician Practice/Performance?

Re-measure the specific practice after the educational intervention to assure desired result. (EXAMPLE)

Measure performance after CME/activity/s– Method

Direct Measurement of performance Chart Audit

– Results: 75% of the prescriptions written by physicians for their CHF patients were ACE I, when indicated

Role for CME Provider - 1

Ask the right questions when planning CME activity. – Are there data to reflect a need and provide a

baseline for performance of individual/organization?– What result do you want from the CME intervention?

Work directly with your defined population, such as a physician practice, or your institution, to identify the gap in practice/health status.

Partner with units that have data that reflect need for change in practice/health status and that continually re-measure performance. (P&T Comm., QI/PI Comm.)

Role for CME Provider - 2

Learn about performance measurement and measure and share results

Learn to do chart audits, or teach physician office staff to conduct audits, or gather other data, for assessment of performance/health status as a standard procedure.

Role for CME Provider - 3

Make physicians Aware of gap, Agree on need to change, Adopt the new practice and Adhere to new practice.(Committee members or Influential physicians)

Use educational practices of – Predisposing – Create awareness of need to change– Enabling – Provide tools (education) to make change– Reinforcing – Follow up to support needed change

CME Profession Competencies

Adult/Organizational Learning Principles

Educational Interventions

Performance Measurement

Systems Thinking

Partnering

Leadership

Administration/ Management

Self Assessment and Lifelong Learning

CME Profession Competencies

Performance Measurement – Use appropriate data to assess educational and administrative performance of the CME Program

Partnering – Identify and collaborate with key partners and stakeholders in accomplishing their CME mission

Systems Thinking – Continually assess individual and organizational performance and make improvements through relevant learning experiences.

Lessons Learned

Accurate Needs Assessment Data a Necessary Foundation For Outcomes Evaluation

Re-measure Interventions Use 3 CME Profession

Competencies in Outcomes Evaluation

– Performance Measurement – Partnering– Systems Thinking

CME must provide outcomes evaluation to contribute to effective health care in the future

Role of CME Provider is expanding from a meeting planner to learning facilitator, or learning consultant

DATA, DATA, DATA

Other Lessonsor

QUESTIONS?

What should YOU do when you return to work in your setting/venue?

Identify one area where performance or health status is necessary for improvement

Ask the right questions in the planning process to assure need and desired results

Re-measure performance or health status after educational intervention/s to document results

Develop at least one new competency Promote the effectiveness and value of CME

References/Resources

ACME Evaluating Educational Outcomes-An Electronic Workbook for CME Providers www.acme-assn.org/workbook.shtml (Available Spring 2009)

CMExchange AMA Physician Consortium on Performance

Development, www.ama-assn/ama/pub/category/2946.html