Rapid Response Evaluation for Physician Office Electronic...
Transcript of Rapid Response Evaluation for Physician Office Electronic...
Rapid Response Evaluation for Physician Office
Electronic Medical Records
Morgan Price, MD, PhD, CCFPFrancis Lau, PhD, eHealth Chair
1Tuesday, July 13, 2010
Thank you to the team• Dr. Francis Lau the eHealth Chair
• eHealth Observatory Team:
• Tyrone Austen
• Jesdeep Bassi
• Heidi Bell
• UBC Enhanced Skills Trainees:
• Dr. Jeanette Boyd
• Dr. Jame Lai
• Dr. Colin Partridge
2Tuesday, July 13, 2010
Who am I?
3Tuesday, July 13, 2010
Who am I?Lead Researcher at eHealth Observatory
3Tuesday, July 13, 2010
Who am I?Lead Researcher at eHealth Observatory
Family Doctor: Cool Aid CHC
3Tuesday, July 13, 2010
Who am I?Lead Researcher at eHealth Observatory
Family Doctor: Cool Aid CHC
UBC Family Medicine Lead Faculty, Informatics
3Tuesday, July 13, 2010
Who am I?Lead Researcher at eHealth Observatory
Family Doctor: Cool Aid CHC
UBC Family Medicine Lead Faculty, Informatics
Adjunct Faculty, Comp Sci
3Tuesday, July 13, 2010
Who am I?Lead Researcher at eHealth Observatory
Family Doctor: Cool Aid CHC
UBC Family Medicine Lead Faculty, Informatics
Adjunct Faculty, Comp Sci
Informatics Consultant
3Tuesday, July 13, 2010
Ground Rules
Please Interrupt
Me with Questions!
4Tuesday, July 13, 2010
Who are you?
5Tuesday, July 13, 2010
My Assumptions(about you)
• You have some knowledge in terms of EMR functionality.
• Wanting hands-on / practical advice on evaluation of adoption.
• Want discussion, not lecture
• You are a mix of providers and IT / informatics professionals.
6Tuesday, July 13, 2010
Anything specific you want to discuss today?
7Tuesday, July 13, 2010
How many of you attended this morning’s
workshop?
8Tuesday, July 13, 2010
Objectives
• Appreciate Current State on EMR adoption evaluation
• Appreciate EMR adoption is a process
• Review formative evaluation methods that can help practices adopt EMR successfully.
9Tuesday, July 13, 2010
Activities
• Review Background from the Literature
• Overview of our evaluation framework
• Overview of the methods and tools
• Trial one of the tools
• Review Two Case Studies
• Discuss Discuss Discuss
10Tuesday, July 13, 2010
EMR Background Literature
11Tuesday, July 13, 2010
We are completing a systematic review of
EMR benefits in office practice.
12Tuesday, July 13, 2010
Question:What are the impacts of
EMR systems on physician office practices?
13Tuesday, July 13, 2010
Methods
Main sources MEDLINECINAHL
Supplemental searchesPublication years: 2000-2009CriteriaReviewers
14Tuesday, July 13, 2010
15Tuesday, July 13, 2010
15,847 Papers
15Tuesday, July 13, 2010
15,847 Papers
Analytic Studies22
Descriptive Studies24
Surveys26
15Tuesday, July 13, 2010
26 published surveys
9 (35%) clearly positive outcomes
14 (54%) reported mixed outcomes
2 (8%) reported negative outcomes
1 had ambiguous results
16Tuesday, July 13, 2010
72% of indicators were for User Satisfaction
System Quality
Information Quality
Service Quality
Use
User Satisfaction
Net Benefits
Quality
Access
Productivity
17Tuesday, July 13, 2010
Analytic Studies
18Tuesday, July 13, 2010
Country # of Studies
% of Total
Netherlands 5 23%
United Kingdom 5 23%
United States 5 23%
Canada 2 9%
Australia 2 9%
New Zealand 2 9%
Italy 1 5%
Total 22 100%
Location of Studies:
50% Europe
32% North America
18% Australia & NZ
19Tuesday, July 13, 2010
Quality of Studies
Study Design # of Studies Strong Moderate Weak % of Total
Randomized 10 6 4 0 67%
Quasi-Experimental 4 0 4 0 20%
Observational 8 2 4 2 13%
Total 22 8 12 2 100%
64% used an experimental or quasi-experimental design
90% had a moderate to strong quality score
20Tuesday, July 13, 2010
Function # of Studies
% of Total
Prescribing 7 32%Prevention 4 18%
CDM 4 18%
Test ordering 2 9%Recordkeeping 2 9%
Other 2 9%Referrals 1 5%
Total 22 100%
Prevention studies were primarily concerned with CVD screening
CDM studies focused on diabetes, arthritis, and CVD
21Tuesday, July 13, 2010
90% of measures were focused on Net Benefits
System Quality
Information Quality
Service Quality
Use
User Satisfaction
Net Benefits
Quality
Access
Productivity
22Tuesday, July 13, 2010
NET BENEFITSNET BENEFITSNET BENEFITSNET BENEFITSNET BENEFITSNET BENEFITSNET BENEFITSNET BENEFITS
QualityQualityQuality AccessAccess ProductivityProductivityProductivity
Patient Safety Appropriateness & effectiveness Health outcomes
Patient & provider access
to services
Patient & caregiver
participationEfficiency Care
coordination Net costs
1 29 5 0 1 4 0 2
There are 42 outcome measures83% quality, 14% productivity, 2% access69% appropriateness & effectiveness
Adherence and compliance with guidelinesRates of testing
23Tuesday, July 13, 2010
Function # of Studies Positive OutcomesPrescribing 7 3
Prevention 4 3CDM 4 2
Test ordering 2 2Recordkeeping 2 0
Other 2 0Referrals 1 1
Total 22 11
24Tuesday, July 13, 2010
Function # of Studies Positive OutcomesPrescribing 7 3
Prevention 4 3CDM 4 2
Test ordering 2 2Recordkeeping 2 0
Other 2 0Referrals 1 1
Total 22 11
Only 2 studies looked at clinical outcomes (no positive findings)
24Tuesday, July 13, 2010
Description of the Descriptive studies
In ProgressIn Progress
25Tuesday, July 13, 2010
There is some evidence of
mixed benefits...lots of room for
improvement
26Tuesday, July 13, 2010
We wanted to add to EMR evaluation.
27Tuesday, July 13, 2010
Overview of our Framework
28Tuesday, July 13, 2010
Scanning
LaboratoryResults
DI ReportsImages
Medications
OtherDocuments
ElectronicFeeds In
ElectronicFeeds Out?
Paper Charts
Paper / FaxOut to others
EMR
Other ElectronicRepositories
(Public and Private)
How to Evaluate?
29Tuesday, July 13, 2010
Scanning
LaboratoryResults
DI ReportsImages
Medications
OtherDocuments
ElectronicFeeds In
ElectronicFeeds Out?
Paper Charts
Paper / FaxOut to others
EMR
Other ElectronicRepositories
(Public and Private) How to Evaluate?
30Tuesday, July 13, 2010
Scanning
LaboratoryResults
DI ReportsImages
Medications
OtherDocuments
ElectronicFeeds In
ElectronicFeeds Out?
Paper Charts
Paper / FaxOut to others
EMR
Other ElectronicRepositories
(Public and Private) How to Evaluate?
How to Provide
Feedback?
30Tuesday, July 13, 2010
We wanted something timely, comprehensible, and useful for clinicians.
31Tuesday, July 13, 2010
We wanted something that could also aid in adoption / impact.
32Tuesday, July 13, 2010
We considered the premise that stepwise adoption limits change
and maintains productivity.
33Tuesday, July 13, 2010
From Stead, 2007
Time
Clin
ical
Tea
m P
erfo
rman
ce
ResultsViewer
CommunicationTools
CPOE
ClinicalDocumentation
34Tuesday, July 13, 2010
We assumed that offices adopt functions more organically over time.
35Tuesday, July 13, 2010
Time
Clin
ical
Per
form
ance
Increasing Performancewith Practice Improvement
36Tuesday, July 13, 2010
Adoption increases over time when change
is manageable and improvements are
perceived.
37Tuesday, July 13, 2010
Adoption may fail where benefit not seen.
38Tuesday, July 13, 2010
Time
Clin
ical
Per
form
ance
39Tuesday, July 13, 2010
Support change by showing improvements
and by providing guideposts.
40Tuesday, July 13, 2010
Our framework is built on two sources:
HIMSS & IOM.
41Tuesday, July 13, 2010
HIMSS Analytics
42Tuesday, July 13, 2010
HIMSS is well know for its 7 Levels for Hospitals
(and it has 5 levels for ambulatory care).
43Tuesday, July 13, 2010
44Tuesday, July 13, 2010
0 Paper Charts for Clinical Information
44Tuesday, July 13, 2010
1
0
Online access to reference materials
Paper Charts for Clinical Information
44Tuesday, July 13, 2010
2
1
0
Hybrid Electronic - Paper Chart
Online access to reference materials
Paper Charts for Clinical Information
44Tuesday, July 13, 2010
3
2
1
0
Electronic Medical Record Used (more passive record)
Hybrid Electronic - Paper Chart
Online access to reference materials
Paper Charts for Clinical Information
44Tuesday, July 13, 2010
4
3
2
1
0
Electronic Chart with Advanced Point of Care Clinical Decision Support
Electronic Medical Record Used (more passive record)
Hybrid Electronic - Paper Chart
Online access to reference materials
Paper Charts for Clinical Information
44Tuesday, July 13, 2010
5
4
3
2
1
0
EMR Integrated within Health SystemPoint of Reflection CDSS
Electronic Chart with Advanced Point of Care Clinical Decision Support
Electronic Medical Record Used (more passive record)
Hybrid Electronic - Paper Chart
Online access to reference materials
Paper Charts for Clinical Information
44Tuesday, July 13, 2010
Institute of Medicine
45Tuesday, July 13, 2010
IOM created a report on the Key Capabilities
of an EHRS(IOM 2003)
46Tuesday, July 13, 2010
IOM focused on care delivery functions and not
infrastructure / data standards.
47Tuesday, July 13, 2010
IOM - 8 Core Functions
48Tuesday, July 13, 2010
IOM - 8 Core Functions
Health Information & Data
Results Management
Order Entry / Order Management
Decision Support
Electronic Communication
Patient Support
Administrative Processes
Reporting & Population Health
48Tuesday, July 13, 2010
IOM - 8 Core Functions
Health Information & Data
Results Management
Order Entry / Order Management
Decision Support
Electronic Communication
Patient Support
Administrative Processes
Reporting & Population Health
48Tuesday, July 13, 2010
IOM - 8 Core Functions
Health Information & Data
Results Management
Order Entry / Order Management
Decision Support
Electronic Communication
Patient Support
Administrative Processes
Reporting & Population Health
48Tuesday, July 13, 2010
IOM - 8 Core Functions
Health Information & Data
Results Management
Order Entry / Order Management
Decision Support
Electronic Communication
Patient Support
Administrative Processes
Reporting & Population Health
48Tuesday, July 13, 2010
IOM - 8 Core Functions
Health Information & Data
Results Management
Order Entry / Order Management
Decision Support
Electronic Communication
Patient Support
Administrative Processes
Reporting & Population Health
48Tuesday, July 13, 2010
IOM - 8 Core Functions
Health Information & Data
Results Management
Order Entry / Order Management
Decision Support
Electronic Communication
Patient Support
Administrative Processes
Reporting & Population Health
48Tuesday, July 13, 2010
IOM - 8 Core Functions
Health Information & Data
Results Management
Order Entry / Order Management
Decision Support
Electronic Communication
Patient Support
Administrative Processes
Reporting & Population Health
48Tuesday, July 13, 2010
IOM - 8 Core Functions
Health Information & Data
Results Management
Order Entry / Order Management
Decision Support
Electronic Communication
Patient Support
Administrative Processes
Reporting & Population Health
48Tuesday, July 13, 2010
We created a complex matrix...IOM Features
HIM
SS L
evel
s
49Tuesday, July 13, 2010
and ended up with a simple graph
0
1
2
3
4
5
Health Inform
ation
Order E
ntry
Results M
anagement
Decision S
upport
Electronic C
omm
unication
Patient S
upport
Scheduling B
illing
Practice R
eflection
50Tuesday, July 13, 2010
that can show changes over time
0
1
2
3
4
5
Health Inform
ation
Order E
ntry
Results M
anagement
Decision S
upport
Electronic C
omm
unication
Patient S
upport
Scheduling B
illing
Practice R
eflection
51Tuesday, July 13, 2010
Next, we developed tools to assess adoption
52Tuesday, July 13, 2010
Next, we developed tools to assess adoption
Impact Assessment (I)
EMR Adoption (I)
Workflow Modelling / Rx Functions (I)
Usability Benchmarking (O)
Practice Reflection (G)
EMR System Analysis (I / S)
52Tuesday, July 13, 2010
Next, we developed tools to assess adoption
Impact Assessment (I)
EMR Adoption (I)
Workflow Modelling / Rx Functions (I)
Usability Benchmarking (O)
Practice Reflection (G)
EMR System Analysis (I / S)
52Tuesday, July 13, 2010
Next, we developed tools to assess adoption
Impact Assessment (I)
EMR Adoption (I)
Workflow Modelling / Rx Functions (I)
Usability Benchmarking (O)
Practice Reflection (G)
EMR System Analysis (I / S)
52Tuesday, July 13, 2010
Next, we developed tools to assess adoption
Impact Assessment (I)
EMR Adoption (I)
Workflow Modelling / Rx Functions (I)
Usability Benchmarking (O)
Practice Reflection (G)
EMR System Analysis (I / S)
52Tuesday, July 13, 2010
Next, we developed tools to assess adoption
Impact Assessment (I)
EMR Adoption (I)
Workflow Modelling / Rx Functions (I)
Usability Benchmarking (O)
Practice Reflection (G)
EMR System Analysis (I / S)
52Tuesday, July 13, 2010
Next, we developed tools to assess adoption
Impact Assessment (I)
EMR Adoption (I)
Workflow Modelling / Rx Functions (I)
Usability Benchmarking (O)
Practice Reflection (G)
EMR System Analysis (I / S)
52Tuesday, July 13, 2010
Time
53Tuesday, July 13, 2010
Time
Focus on Use
53Tuesday, July 13, 2010
All resources are available at
ehealth.uvic.ca
(version 2.0 coming)
54Tuesday, July 13, 2010
METHOD: we travel
on site, tools in hand.
55Tuesday, July 13, 2010
Two researchers Three days
Focus Group
Feedback Focus Group
Interview
Interview
Interview
Interview
Observation
Observation
Interview
Interview
Usability
Usability
Interview
Interview
56Tuesday, July 13, 2010
EMR System Assessment
57Tuesday, July 13, 2010
Should be completed before site visits.
58Tuesday, July 13, 2010
Evaluation of functionality to provide
“ceiling” on scores.
59Tuesday, July 13, 2010
Necessary to know what the clinic can and
cannot do.
60Tuesday, July 13, 2010
Aligned our evaluation with existing functional
requirements.
61Tuesday, July 13, 2010
Our focus was on Medication
Management
62Tuesday, July 13, 2010
Impact Assessment Interview
63Tuesday, July 13, 2010
Two part interview
• Part 1
• Impact Assessment
• Mainly open-ended questions
• Part 2 (Pare)
• Focused on Change Management / Deployment
• 23 questions, 5 point scale
64Tuesday, July 13, 2010
Impact Assessment Examples
• What do you think about the quality of the system in terms of functionality and performance?
• What opportunities do you see with the use of the EMR as compared to how things were previously done?
65Tuesday, July 13, 2010
EMR Deployment Examples
• 1-5 Scale of Agreement to statements like:
• There was alignment of the system with local practices and processes.
• There were positive attitudes on the part of project team members.
• It was a small and simple project.
66Tuesday, July 13, 2010
EMR Adoption Interview
67Tuesday, July 13, 2010
EMR Adoption Interview
• Covers all eight functional areas
• Currently an interview
• Could be survey, after validation
• Each question scored based on current use
• Developing MOA and MD versions.
68Tuesday, July 13, 2010
EMR Adoption Interview is a good level of detail for an
office.
69Tuesday, July 13, 2010
We are going to trial this 25 question interview today.
70Tuesday, July 13, 2010
Activity: (45 min)
EMR Adoption Interview
71Tuesday, July 13, 2010
Instructions• Act as a clinician who is using an EMR.
• Answer as a single user.
• Try to be realistic.
• Score to the highest score (if multiple options correct)
• Use judgement if not quite fitting the five levels precisely.
• Comment on the challenges
72Tuesday, July 13, 2010
Break(and scoring)
73Tuesday, July 13, 2010
Debrief the Interview - what did you find?
74Tuesday, July 13, 2010
SCORES
75Tuesday, July 13, 2010
One
0
1
2
3
4
5
Health Inform
ation
Order E
ntry
Results M
anagement
Decision S
upport
Electronic C
omm
unication
Patient S
upport
Scheduling B
illing
Practice R
eflection
76Tuesday, July 13, 2010
Two
0
1
2
3
4
5
Health Inform
ation
Order E
ntry
Results M
anagement
Decision S
upport
Electronic C
omm
unication
Patient S
upport
Scheduling B
illing
Practice R
eflection
77Tuesday, July 13, 2010
Three
0
1
2
3
4
5
Health Inform
ation
Order E
ntry
Results M
anagement
Decision S
upport
Electronic C
omm
unication
Patient S
upport
Scheduling B
illing
Practice R
eflection
78Tuesday, July 13, 2010
Anything Unclear
79Tuesday, July 13, 2010
Anything we missed?
80Tuesday, July 13, 2010
Any surprises from the scores?
81Tuesday, July 13, 2010
Workflow Modelling / Medication
Management Interview
82Tuesday, July 13, 2010
Explores detailed processes of medication
management.
83Tuesday, July 13, 2010
Takes a broad view of the workflows of
medication management.
84Tuesday, July 13, 2010
85Tuesday, July 13, 2010
Distilled to a set of questions with answers
that map to the five levels.
86Tuesday, July 13, 2010
Some questions BC specific.
(Special Authority)
87Tuesday, July 13, 2010
These questions also
tend to educate.
88Tuesday, July 13, 2010
Usability Benchmarking
89Tuesday, July 13, 2010
Observation (in this case) of
prescribing activity.
90Tuesday, July 13, 2010
We developed six standardized cases.
91Tuesday, July 13, 2010
Standardized cases allow for comparison at three levels: user, clinic, & EMR.
92Tuesday, July 13, 2010
Feedback: early to users, later to users and
to vendor.
93Tuesday, July 13, 2010
Practice ReflectionFocus Group
94Tuesday, July 13, 2010
First, we open the floor to their reflection, as a
group.
95Tuesday, July 13, 2010
We provide feedback and recommendations
96Tuesday, July 13, 2010
Focus Group reviews findings, agrees to
priorities and discusses how to implement.
97Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
Finally, we repeat the study.
98Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
and look for change over time.
99Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
and look for change over time.
99Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
and look for change over time.
99Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
and look for change over time.
99Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
and look for change over time.
99Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
and look for change over time.
99Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
and look for change over time.
99Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
and look for change over time.
99Tuesday, July 13, 2010
240 6 12 18
Months
Adoption
and look for change over time.
99Tuesday, July 13, 2010
Two Case Studies
100Tuesday, July 13, 2010
Case Study 1
101Tuesday, July 13, 2010
Here are the scores.
0
1
2
3
4
5
Health Inform
ation
Order E
ntry
Results M
anagement
Decision S
upport
Electronic C
omm
unication
Patient S
upport
Scheduling B
illing
Practice R
eflection
102Tuesday, July 13, 2010
Change Management - variable, based on role.
103Tuesday, July 13, 2010
Prescription Management
• Variable use of EMR
• No alerting
• Not using advanced features
104Tuesday, July 13, 2010
Usability
• Multiple errors from system / GUI
• Users did not use tools efficiently
• Drug selection issues (generic / brand)
• Complexity for newer users
• icons not clear
• Complex prescriptions not handled well.
105Tuesday, July 13, 2010
Other Issues
• Participants quick to highlight challenges.
• Many are relative complaints:
• it is slower now
• scanning takes way more mouse clicks than it used to.
• Technical glitches: freezing
• Training issues.
106Tuesday, July 13, 2010
They didn’t complain about what they didn’t know.
107Tuesday, July 13, 2010
Case Study 2(more briefly)
108Tuesday, July 13, 2010
0
1
2
3
4
5
Health Inform
ation
Order E
ntry
Results M
anagement
Decision S
upport
Electronic C
omm
unication
Patient S
upport
Scheduling B
illing
Practice R
eflection
Here are the scores.
109Tuesday, July 13, 2010
Let’s Compare the Two Cases110Tuesday, July 13, 2010
Case 1
0
1
2
3
4
5
Health Inform
ation
Order E
ntry
Results M
anagement
Decision S
upport
Electronic C
omm
unication
Patient S
upport
Scheduling B
illing
Practice R
eflection
111Tuesday, July 13, 2010
0
1
2
3
4
5
Health Inform
ation
Order E
ntry
Results M
anagement
Decision S
upport
Electronic C
omm
unication
Patient S
upport
Scheduling B
illing
Practice R
eflection
Case 2
112Tuesday, July 13, 2010
Who implemented when?
113Tuesday, July 13, 2010
Case 2 implemented 10 years before Case 1
114Tuesday, July 13, 2010
Why so little difference?
115Tuesday, July 13, 2010
User Challenges
116Tuesday, July 13, 2010
I know how paper works...
117Tuesday, July 13, 2010
I didn’t know what I didn’t know
118Tuesday, July 13, 2010
Short term fixes have long term implications
119Tuesday, July 13, 2010
Terminology, shmerminology - let me write down what I want.
120Tuesday, July 13, 2010
System Challenges
121Tuesday, July 13, 2010
Group1,000s
Classification10,000s
Reference Terminology100,000s
122Tuesday, July 13, 2010
“Flexible Data models”“Add your own terminologies”
123Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
From my office...
124Tuesday, July 13, 2010
How can I see my practice with data like that?
125Tuesday, July 13, 2010
How can we share data in the future?
126Tuesday, July 13, 2010
127Tuesday, July 13, 2010
Question for you:What is missing from
our tools?
128Tuesday, July 13, 2010
Materials available atehealth.uvic.ca
129Tuesday, July 13, 2010