Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health...
-
date post
19-Dec-2015 -
Category
Documents
-
view
219 -
download
2
Transcript of Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health...
![Page 1: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/1.jpg)
Clinical Decision Support Systems
Syed Tirmizi, M.D.Medical Informatician
Veterans Health Administration
![Page 2: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/2.jpg)
Clinical Decision Support Systems
• Definition (What)
• Business case (Why)
• Use Cases (How)
• Usability testing & Evaluations
![Page 3: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/3.jpg)
Decision Support Systems
Decision support systems are a class of computer-based information systems including knowledge based systems that support decision making activities.
-Wikipedia
![Page 4: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/4.jpg)
Decision Support Systems• A passive DSS is a system that aids the process of
decision making, but that cannot bring out explicit decision suggestions or solutions.
• An active DSS can bring out such decision suggestions or solutions.
• A cooperative DSS allows the decision maker (or its advisor) to modify, complete, or refine the decision suggestions provided by the system, before sending them back to the system for validation.
Haettenschwiler
![Page 5: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/5.jpg)
Clinical Decision Support Systems
• computer software employing a knowledge base designed for use by a clinician involved in patient care, as a direct aid to clinical decision making
• a set of knowledge-based tools that are fully integrated with both the clinician workflow components of a computerized patient record, and a repository of complete and accurate data
• providing clinicians or patients with clinical knowledge and patient-related information, intelligently filtered and presented at appropriate times, to enhance patient care
Clinical Decision Support in Electronic Prescribing: Recommendations and an Action Plan
Report of the Joint Clinical Decision Support WorkgroupJONATHAN M. TEICH, MD, PHD, JEROME A. OSHEROFF, MD, ERIC A. PIFER, MD, DEAN F.SITTIG, PHD, ROBERT A. JENDERS, MD, MS, THE CDS EXPERT REVIEW PANEL
J Am Med Inform Assoc. 2005;12:365–376.
![Page 6: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/6.jpg)
Patient Safety & Quality Gaps Acknowledged
• “98,000 Hospital Patients Die Yearly Because of Adverse Events” – (IOM, 1999)
• “Virtually Every Patient Experiences a Gap Between the Best Evidence and the Care They Receive” – (IOM, 2001)
![Page 7: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/7.jpg)
Outpatient Adverse Drug Events
• Overall— 25% of outpatients incurred an ADE— 39% were preventable— Antidepressants and antihypertensives were
largest contributors
• Elderly (over 65)— Adverse Events in 5% of population per year— 28% preventable
Gandhi et al, NEJM 2003;348(16):1556-1564 Gurwitz et al, JAMA 2003;289:1107-16
![Page 8: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/8.jpg)
Chances of Receiving Appropriate Preventive
Care is about 50%-NEJM
![Page 9: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/9.jpg)
Employer/Payor business case for CDS - Diabetes
• Estimated avg $21,000/year per diabetic employee in absenteeism, disability and medical costs (study of 6 employers with 375,000 employees
• Glycemic control is associated with $1000-$2000 medical costs savings/year to payor
• Currently, we are “reimbursed” to measure HgA1c annually (captured claim for test ordered)
• Will soon be reimbursed for maintaining control through test result surveillance, goal is < 7
Tonya Hongsermeier, MD, MBA
Partners Healthcare Systems
![Page 10: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/10.jpg)
Knowledge Processing Required for Care Delivery
• Medical literature doubling every 19 years— Doubles every 22 months for AIDS care
• 2 Million facts needed to practice • Genomics, Personalized Medicine will increase
the problem exponentially• Typical drug order today with decision support
accounts for, at best, Age, Weight, Height, Labs, Other Active Meds, Allergies, Diagnoses
• Today, there are 3000+ molecular diagnostic tests on the market, typical HIT systems cannot support complex, multi-hierarchical chaining clinical decision support
Covell DG, Uman GC, Manning PR. Ann Intern Med. 1985 Oct;103(4):596-9
![Page 11: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/11.jpg)
Drilling for the Best Information
Cochrane LibraryEB Practice Guideline
Specialty-specificPOEMs
Best Evidence
Clinical EvidenceClinical Inquiries
Reviews: Textbooks, Up-to-Date, 5-Minute Clinical Consult
Use
fuln
ess
Medline
![Page 12: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/12.jpg)
Clinical Clinical requirementsrequirements
Diabetes Patient Dialog for processing multiple reminders:
• Diabetic Foot Care Education• Diabetic Foot Exam• Diabetic Eye Exam• Recommended Labs• Other Health Activities
Acquisition of health data beyond care delivered exclusively through VHA
Standardized Data Elements
Links Reminder
With Actions
With Documentation
![Page 13: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/13.jpg)
![Page 14: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/14.jpg)
![Page 15: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/15.jpg)
![Page 16: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/16.jpg)
![Page 17: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/17.jpg)
![Page 18: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/18.jpg)
![Page 19: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/19.jpg)
![Page 20: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/20.jpg)
![Page 21: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/21.jpg)
![Page 22: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/22.jpg)
Suggest Use of Thiazide
• Set up the reminder dialog so that if the patient is a reasonable candidate for a thiazide and not currently on one, then suggest use of a thiazide.
• Suppressed by Cr>2.0, Calcium>10.2, Na+<136 or allergy.
![Page 23: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/23.jpg)
Standard HTN dialog copied from the national reminder
![Page 24: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/24.jpg)
Insert section at the top if the patient is a candidate for use of a thiazide
![Page 25: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/25.jpg)
![Page 26: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/26.jpg)
“Clinical Reminders”
Performance Measures1. Clinical Reminders
– Real time decision support– Targeted to specific patient cohort– Targeted to specific clinic/clinicians
2. Reminder Dialogs– Standard documentation– Capture of data (HF, encounter data, etc)
3. Reminder Reports– Performance improvement/scheduled feedback– Identification of best practices– Targeting low scorers for educational intervention– Patient recall if missed intervention
![Page 27: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/27.jpg)
Clinical Reminder Reports
• Multiple Uses for Reminder Reports – Patient care:
• Future Appointments– Which patients need an intervention?
• Past Visits– Which patients missed an intervention?
• Action Lists
• Inpatients– Which patients need an intervention prior to discharge?
![Page 28: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/28.jpg)
Clinical Reminder Reports
• Identify patients for case management– Diabetic patients with poor control– Identify patients with incomplete problem lists
• Patients with (+) Hep C test but no PL entry
– Identify high risk patients• on warfarin, amiodarone
– Track annual PPD due (Employee Health)
![Page 29: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/29.jpg)
Clinical Reminder Reports
• Quality Improvement:– Provide feedback (team/provider)– Identify (& share) best practices – Identify under-performers (develop action plan)– Track performance– Implementation of new reminders or new processes– Identify process issues early (mismatch of workload
growth versus staffing)– Provide data for external review (JCAHO)
![Page 30: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/30.jpg)
Clinical Reminder Reports
• Management Tool– Aggregate reports
• Facility / Service
• Team (primary care team)
• Clinic / Ward
– Provider-specific reports• Primary Care Provider
• Encounter location
• If one provider per clinic location
![Page 31: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/31.jpg)
Reminder/Dialogs: Other UsesExamples: Reminder dialogs linked to note title • Present ordering dialogs
– Medications Orders• Sildenafil/levitra (screening for risk factors)• Clopidogrel (Plavix) (updated criteria)
– Discharge Order• Support medication reconciliation (when pharmacists
are not available to review meds)• Gather information for display on Health Summary
– Non VA surgery
![Page 32: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/32.jpg)
Computerized Patient Record System CPRS
• Improve healthcare outcomes• Translate Clinical Practice Guidelines into clinical
activities • Real time decision support for clinicians at point of
care – reminders, alerts – Prevent patient from falling through the cracks– Avoid reliance on memory, vigilance
• Reduce errors (omissions, transcriptions, etc)• Facilitate documentation for performance
measurement and improvement efforts
![Page 33: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/33.jpg)
However
This is NOT about technology…
It is about RESULTS:
• Improved Health Care Quality
• Improved Health Outcomes
![Page 34: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/34.jpg)
How Do We Compare to non-VA Providers? VHA Continues to exceed HEDIS in the vast majority of 17
common measures
CLINICAL PERFORMANCE INDICATOR
VA FY 05HEDIS Commercial 2004
HEDIS Medicare 2004
HEDIS Medicaid 2004
Breast cancer screening 86% 73% 74% 54%
Cervical cancer screening 92% 81% Not Reported 65%
Colorectal cancer screening 76% 49% 53% Not Reported
LDL Cholesterol < 100 after AMI, PTCA, CABG
Not Reported 51% 54% 29%
LDL Cholesterol < 130 after AMI, PTCA, CABG
Not Reported
68% 70% 41%
Beta blocker on discharge after AMI 98% 96% 94% 85%
Hypertension: BP <= 140/90 most recent visit
77% 67% 65% 61%
Follow-up after Hospitalization for Mental Illness (30 days)
70% 76% 61% 55%
HEDIS = Health Plan Employer Data & Information Set From the National Committee on Quality Assurance (NCQA)
![Page 35: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/35.jpg)
How Do We Compare to non-VA Providers? VHA Continues to exceed HEDIS in the vast majority of
17 common measuresCLINICAL PERFORMANCE INDICATOR
VA FY 05HEDIS Commercial 2004
HEDIS Medicare 2004
HEDIS Medicaid 2004
Diabetes: HgbA1c done past year 96% 87% 89% 76%
Diabetes: Poor control HbA1c > 9.0% (lower is better)
17% 31% 23% 49%
Diabetes: Cholesterol (LDL-C) Screening
95% 91% 94% 80%
Diabetes: Cholesterol (LDL-C) controlled (<100)
60% 40% 48% 31%
Diabetes: Cholesterol (LDL-C) controlled (<130)
82% 65% 71% 51%
Diabetes: Eye Exam 79% 51% 67% 45%
Diabetes: Renal Exam 66% 52% 59% 47%
CLINICAL PERFORMANCE INDICATOR VA FY 2005
HEDIS Commercial 2004
HEDIS Medicare 2004
BRFSS 2004
Immunizations: influenza, (note patients age groups)
75% (65 and older or high risk)
39% (50-64)
75% (65 and older)
68% (65 and older)
Immunizations: pneumococcal, (note patients age groups)
89% (all ages at risk)
Not Reported Not Reported65% (65 and older)
![Page 36: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/36.jpg)
FY99-04 Changes in Total, Major and Minor Age-Adjusted Amputation Rates Among
Patients With Diabetes
0
1
2
3
4
5
6
7
8
9
Overall 7.94 6.24 5.42 4.53 4.4 4.04
Major 3.61 2.78 2.4 1.95 1.84 1.72
Minor 4.33 3.46 3.03 2.59 2.55 2.32
1999 2000 2001 2002 2003 2004
Am
puta
tions
per
100
0 pa
tien
ts
![Page 37: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/37.jpg)
Pneumococcal Vaccination Rates in VHA
0
20
40
60
80
100
FY 95 4th Qtr97
4th Qtr98
FY 99 FY 00 CHG FY01* FY02 FY03
Pe
rce
nt
Va
cc
ina
ted
VHA Healthy People 2000 Iowa 99* NHIS
`
•Iowa: Petersen, Med Care 1999;37:502-9. >65/ch dz•HHS: National Health Interview Survey, >64
--BRFSS--
--BRFSS 90th--
![Page 38: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/38.jpg)
Performance has improvedScreening for Colon Cancer
5360 64 67
72 74 77
0
1020
30
4050
60
7080
90
FY00 FY01 FY02 FY03 FY04 FY05 FY06
Colon Cancer Screen
![Page 39: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/39.jpg)
Timely Eye Exam for Patients with Diabetes
4455
62 62 6166
72 7580 80
85
0
10
20
30
40
50
60
70
80
90
100
FY 95 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06
![Page 40: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/40.jpg)
Pneumococcal Immunizations
26
61
7377
81 84 8185 87 89 89
0
10
20
30
40
50
60
70
80
90
100
VHA (High risk or >= 65yrs
Changed to include refusals
as failures
![Page 41: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/41.jpg)
Outcomes have improved
• Increased rates of pneumococcal vaccination over past 5 years has averted over 4000 deaths nationally in VA patients with lung disease
• Diabetic complications markedly decreased – amputations, peripheral neuropathy, visual impairment and loss
![Page 42: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/42.jpg)
Improved Outcomes
Productive Interactions
DeliverySystemDesign
DecisionSupport
VistASelf-Management
Support
Health SystemResources and Policies
Community Organization of Health Care
Informed,Empowered Patient
and Family
Prepared,Proactive Practice
Team
The Chronic Disease Care Model
Patient-Centered
Coordinated
Timely and Efficient
Evidence-based and Safe
My HealthMy HealtheeVetVet
![Page 43: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/43.jpg)
Highest Quality of Care For Patients with Diabetes in VA “Diabetes processes of care and 2 of 3 intermediate outcomes were better for patients in the VA system than for patients in commercial managed care.”
Annals of Internal Medicine, August 17, 2004
![Page 44: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/44.jpg)
Highest Quality of Care For Patients in VA Measured Broadly
“Patients from the VHA received higher-quality care according to a broad measure. Differences were greatest in areas where the VHA has established performance measures and actively monitors performance.”
Annals of Internal Medicine, December 21, 2004
![Page 45: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/45.jpg)
Guideline-Based Decision Support for Hypertension with
ATHENA DSS
Implementation &
Evaluation
Mary K. Goldstein, MD
![Page 46: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/46.jpg)
Developing a Model Program
To Provide a Model Program that can be extended to other clinical areas
They selected hypertension as a model for guideline implementation because…
• Hypertension is highly prevalent in adult medical practice
• There are excellent evidence-based guidelines for management
• There is also evidence that the guidelines are not well-followed– a big ‘improvability gap’ in IOM terms
• Steinman, M.A., M.A. Fischer, M.G. Shlipak, H.B. Bosworth, E.Z. Oddone, B.B. Hoffman and M.K. Goldstein, Are Clinicians Aware of Their Adherence to
Hypertension Guidelines? Amer J. Medicine 117:747-54, 2004.
![Page 47: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/47.jpg)
What the Clinician Sees…
![Page 48: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/48.jpg)
ATHENA Hypertension Advisory:BP- Prescription Graphs
Goldstein, M. K. and B. B. Hoffman (2003). Graphical Displays to Improve Guideline-Based Therapy of Hypertension. Hypertension Primer. J. L. Izzo, Jr and H. R. Black. Baltimore, Williams & Wilkins.
![Page 49: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/49.jpg)
ATHENA HTN Advisory
BP targets
Primaryrecommendation
Drugrecommendation
![Page 50: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/50.jpg)
ATHENA HTN Advisory: More Info
![Page 51: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/51.jpg)
What is ATHENA DSS?• Automated decision support system (DSS)
– Knowledge-based system automating guidelines• Built with EON technology
– For patients with primary hypertension who meet eligibility criteria
• Patient specific information and recommendations at the point of care
• Purpose is to improve hypertension control and prescription concordance with guidelines
•Athena in Greek mythology is a symbol of good counsel, prudent restraint, and practical insight
•Proc AMIA 2000
![Page 52: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/52.jpg)
ATHENA Protégé top level
![Page 53: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/53.jpg)
ATHENA Protégé GL management
diagram
![Page 54: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/54.jpg)
ATHENA Clients
AdvisoryClient
EventMonitor
Building ATHENA System From EON Components
SQLPatient
Database
ATHENA Clients
EON Servers
GuidelineInterpreter Advisory
Client
EventMonitor
TemporalMediator
VA CPRSVISTA
Data Converter
nightly data extraction ATHENA
HTNGuideline
KnowledgeBase
Protégé ATHENA GUI
Pre-computedAdvisories
![Page 55: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/55.jpg)
Path to Guideline Adherence
The theoretical model we use for the path to guideline adherence is the “Awareness to Adherence” model, in which the clinician must
– Awareness of guideline– Acceptance of guideline – Adoption of guideline– Adherence to guideline Pathman, D. E., T. R. Konard, et al. (1996). "The Awareness-to-
Adherence Model of the Steps to Clinical Guideline Compliance." Medical Care 34:873-889.
![Page 56: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/56.jpg)
Informatics Support for Clinical Practice Guideline Implementation
Step Facilitators Informatics
Support
AwarenessPriming Activities such as
profiling of baseline performance
Profiling from pharmacy and diagnosis database
Acceptance
Active education such as Academic Detailing;
Clinical Opinion Leaders
Present evidence relevant to patient; allow
opinion leaders to browse knowledge
AdoptionEnabling strategies such as
incorporation into clinic workflow
Integration with existing EMR
AdherenceReinforcing Strategies such as
remindersPoint-of-care patient-
specific advisories
![Page 57: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/57.jpg)
Challenge of Using IT for Quality Improvement
• Technical challenges of using information technology for quality improvement (QI)
– Difficult to integrate new forms of decision support into legacy data systems and electronic record interfaces
– We had many design requirements in order to meet research goals and institutional goals
– A “sociotechnical” challenge to implement
Goldstein, M., R. Coleman, S. Tu, et. Al. Translating Research Into Practice: SocioTechnical Integration of
Automated Decision Support for Hypertension in Three Medical Centers. JAMIA 11: 368-76, 2004.
Available in pubmedcentral
![Page 58: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/58.jpg)
Decision Support for Common Chronic Diseases
The “Field of Dreams” approach to
medical informatics implementations:If you build it, they will come
The physician often seen as wondering about a clinical question and then seeking out decision support:
![Page 59: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/59.jpg)
Some Technical Challenges
• Extracting clinical data from VistA
• Generating a popup window that appears in CPRS– At the right time, in the right clinic settings, for
the right clinician, about the right patient
• Logging data about activity in the system
• Security issues
![Page 60: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/60.jpg)
Some of the Social Challenges
• Clinicians extremely time-pressured in clinic– Strike balance between ease of access to
system and ease of ignoring it
• Enormous variability in comfort with computers– And virtually no training time available
• Disagreements about the guidelines– some want VA GLs, some want JNC
![Page 61: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/61.jpg)
Taking on the Sociotechnical Challenge
• Aligning with institutional goals– Discuss with local stakeholders– VA performance standards and guidelines
• Speaking the language(s)– understanding that different computer worlds are worlds apart
• Identify a bridge person to span the gap between IRMS expertise and non-VA programmers
• Iterative Design– With opportunity for re-design cycles after input from key clinical
staff– Don’t test in clinic prematurely
• Do your offline testing first– Test with typical users, not just early adopters– Recognize need for continual adaptation to our evolving
informatics infrastructure
![Page 62: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/62.jpg)
Evaluation Flowchart
Patient Data
•Eligibility•Target BP•BP under control•Risk group•Drug recommendations•Messages
Evaluation Flowchart
Rules
MD Athena
Comparison
MD versus ATHENA
Martins SB et al Proc AMIA 2006 in press
![Page 63: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/63.jpg)
“Physician Testers” in Clinical Setting
• Project-friendly physicians who test the system in early stages in clinic– Understanding it is not yet complete– Must be prepared to make changes in
response to their comments– Some of these physicians become champions
for the system
• Include clinical managers in early testing
![Page 64: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/64.jpg)
Consensus Conference Calls
• Knowledge updates required in light of newly published clinical trials or new guidelines– Need a knowledge management process for vetting new
material and deciding what will be incorporated– Make this process known to the clinicians who are end-users
(especially local opinion leaders)– Invite local input to the discussion– Encode with a system that allows for easy updating
Goldstein, M.K., B.B. Hoffman, et al, Implementing clinical practice guidelines while taking account of changing evidence: ATHENA
DSS, An easily modifiable decision-support system for managing hypertension in primary care. AMIA Symp: 300-4, 2000.
![Page 65: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/65.jpg)
Ontologies in Clinical Decision Support Applications
Health IT has the potential to improve patient care by adherence to clinical practice guidelines
EON and ATHENA projects demonstrate use of ontologies in clinical decision support applications
![Page 66: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/66.jpg)
EON project NLM-funded project at Stanford (PI: Dr. Musen) Develop methodology, ontologies, and software
components for creating decision-support system for guideline-based care
Use Protégé knowledge-acquisition methodology and tool for construction of Domain concept ontologies Patient information model Guideline knowledge bases
Develop software components that assist clinicians in specific tasks
![Page 67: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/67.jpg)
ATHENA project
Funded by VA Research Service HSR&D Hypothesized that guideline-based interventions in
management of hypertension can Change physicians’ prescribing behavior Change patient outcome
Deployed and evaluated at primary care VA clinics in 9 geographically diverse cities over a 15-month clinical trial
Results Expert clinicians maintain hypertension knowledge base
using Protégé Clinicians interacted with the ATHENA Hypertension
Advisory at 54% of all patient visits Impact on prescribing behavior and patient outcome being
analyzed
![Page 68: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/68.jpg)
Stages in Evaluating Clinical Decision Support Systems 1
Stage
EvalType
EarlyDesignAndDevelop
ExploreFeasibility 2, Reliability,Safetyinformally
IntermedDevelop-ment
MoreFormalTest ofComponents
MoreMatureSystem
Tests in Actual use;Externalreviewers
LargeClinicaltrial,? RCT
WiderImplemen-tation
Post-Fieldingsurveillance
1. Elaborated from Miller RA JAMIA 19962. Use Cases
![Page 69: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/69.jpg)
Stages in Evaluating Clinical Decision Support Systems (CDSS)
Stage
EvalType
EarlyDesignAnddevelop
ExploreFeasibility,Reliability,safetyinformally
IntermedDevelop-ment
MoreFormalTest ofcomponents
MoreMatureSystem
Tests in Actual use;Externalreviewers
LargeClinicaltrial,? RCT
WiderImplemen-tation
Post-Fieldingsurveillance
Goldstein, M.K., et al., Patient Safety in Guideline-Based Decision Support for Hypertension Management: ATHENA DSS. JAMIA, 2002. 9(6 Suppl): p. S11-6.
![Page 70: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/70.jpg)
Patient Safety in New Health IT
New computer systems have potential to reduce errors…
But also potential to create new opportunities for error
![Page 71: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/71.jpg)
Errors due to new Health IT Studies of accidents have shown that new computer
systems can affect human problem solving in ways that contribute to errors data overload
• computer collects and displays information out of proportion to human ability to use it effectively
“automation surprises”• bar code administration unobservable actions
• Goldstein, M.K., et al., Patient safety in guideline-based decision support for hypertension management: ATHENA DSS. J Am Med Inform Assoc, 2002. 9(6 Suppl): p. S11-6.
![Page 72: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/72.jpg)
Charles Friedman and Jeremy Wyatt
![Page 73: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/73.jpg)
Safety Testing Clinical Decision Support Systems
“Before disseminating any biomedical information resource…designed to influence real-world practice decisions…check that it is safe…”
Drug testing in vivo and in vitro Information resource safety testing:
how often it furnishes incorrect advice
Friedman and Wyatt Evaluation Methods in Biomedical Informatics 2006
![Page 74: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/74.jpg)
Stages in Evaluating Clinical Decision Support Systems 1
Stage
EvalType
EarlyDesignAndDevelop
ExploreFeasibility , Reliability,Safetyinformally
IntermedDevelop-ment
MoreFormalTest ofComponents
MoreMatureSystem
Tests in Actual use;Externalreviewers
LargeClinicaltrial,? RCT
WiderImplemen-tation
Post-Fieldingsurveillance
1. Elaborated from Miller RA JAMIA 1996
![Page 75: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/75.jpg)
Stages in Evaluating Clinical Decision Support Systems
Stage
EvalType
EarlyDesignAnddevelop
ExploreFeasibility,Reliability,safetyinformally
IntermedDevelop-ment
MoreFormalTest ofcomponents
MoreMatureSystem
Tests in Actual use;Externalreviewers
LargeClinicaltrial,? RCT
WiderImplemen-tation
Post-Fieldingsurveillance
After Miller RA JAMIA 1996Both initially and after updates
![Page 76: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/76.jpg)
CDSS to Evaluate: ATHENA-HTN
DSS developed using the EON architecture from Stanford Medical Informatics (Musen et al)
Electronic Medical Record System
Patient Data
ATHENA HTN Guideline
Knowledge Base
GuidelineInterpreter/Execution
Engine
SQL Server relational database
![Page 77: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/77.jpg)
Protégé: ontology editorOpen source (http://protege.stanford.edu/)
EON model for practice guidelines Focus for evaluation:
• Eligibility criteria for including patients• Drug reasoning for drug recommendations
Knowledge Base
Tu SW, Musen MA. A Flexible Approach to Guideline Modeling. Proc AMIA Symp; 1999. 420-424
![Page 78: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/78.jpg)
Execution Engine
Applies the guideline as encoded in the knowledge base to the patient’s data
Generates set of recommendations
Tu SW, Musen MA. Proc AMIA Symp; 2000. 863-867
![Page 79: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/79.jpg)
Testing the software for accuracy
![Page 80: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/80.jpg)
“The Art of Software Testing” False definition of testing
E.g., “Testing is the process of demonstrating that errors are not present”
Testing should add value to the program improve the quality
Start with assumption program contains errors A valid assumption for almost any program
“Testing is the process of executing a program with the intent of finding errors.” Purpose of testing: to find as many errors as possible
Myers G, Sandler C, Badgett T, Thomas T. The Art of Software Testing. 2nd Ed. John Wiley & Sons; 2004
![Page 81: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/81.jpg)
Software “Regression Testing” Software updates and changes are
particularly error-prone Changes may introduce errors into a
previously well-functioning system “regress” the system
Desirable to develop a set of test cases with known correct output to run in updated systems before deployment
![Page 82: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/82.jpg)
Stages in Evaluating Clinical Decision Support Systems
Stage
EvalType
EarlyDesignAnddevelop
ExploreFeasibility,Reliability,safetyinformally
IntermedDevelop-ment
MoreFormalTest ofcomponents
MoreMatureSystem
Tests in Actual use;Externalreviewers
LargeClinicaltrial,? RCT
WiderImplemen-tation
Post-Fieldingsurveillance
Both initially and after updates
![Page 83: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/83.jpg)
Clinical Decision Support System Accuracy Testing Phases
![Page 84: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/84.jpg)
Objectives for this phase of testing
Test the knowledge base and the execution engine after an update to the knowledge base and prior to clinical deployment of the updated system to detect errors and improve quality of
system Establish correct output (answers) for
set of test cases
![Page 85: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/85.jpg)
Methods: Overview
Electronic patient data: Test cases
ATHENA-HTN CDSS
ATHENA recommendations
+
Physician
“Rules”
Physician recommendations
Comparison
![Page 86: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/86.jpg)
Physician Evaluator (MD)
Internist with experience in treating hypertension in primary care setting
No previous involvement with ATHENA project
Studied “Rules” and clarified any issues Had “Rules” and original guidelines
available during evaluation of test cases
![Page 87: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/87.jpg)
Elements examined
Patient eligibility Did patient meet ATHENA exclusion criteria?
Drug recommendations List of all possible anti-hypertensive drug
recommendations concordant with guidelines • Drug dosage increases• Addition of new drugs• Drug substitutions
Comments by MD
![Page 88: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/88.jpg)
Comparison Method
Comparing ATHENA vs MD ouput: Automated comparison for
discrepancies Manual review of all cases
Reviewing discrepancies Meeting with physician evaluator Adjudication by third party when
categorizing discrepancies
![Page 89: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/89.jpg)
Successful Test
A successful test is one that finds errors so that you can fix them
![Page 90: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/90.jpg)
Set of “Gold Standard” Test Cases
Iteration between clinician review and system output
Same test cases for bug fixes and elaborations in areas that don’t affect the answers to test cases
Change gold standard answers to test cases when the GL changes i.e., when what you previously thought was
correct is no longer correct
![Page 91: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/91.jpg)
Important features of Offline Testing Method
Challenging CDSS with real patient data Clinician not involved in project: “fresh view”
![Page 92: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/92.jpg)
Additional observation
Difficulty of maintaining a separate “Rules” document that describes encoded knowledge
![Page 93: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/93.jpg)
Benefits of the Offline Testing Offline testing method was successful in identifying
“errors” in ATHENA’s Knowledge base Program boundaries were better defined Updates made improving accuracy before deployment Gold standard answers to test cases Offline Testing of the ATHENA Hypertension Decision Support System
Knowledge Base to Improve the Accuracy of Recommendations.Martins SB, Lai S, Tu SW, Shankar R, Hastings SN, Hoffman BB, Dipilla N, Goldstein MK. AMIA Annu Symp Proc. 2006;539-43
![Page 94: Clinical Decision Support Systems Syed Tirmizi, M.D. Medical Informatician Veterans Health Administration.](https://reader035.fdocuments.us/reader035/viewer/2022062515/56649d265503460f949fda7a/html5/thumbnails/94.jpg)
Stages in Evaluating Clinical Decision Support Systems (CDSS)
Stage
EvalType
EarlyDesignAnddevelop
ExploreFeasibility,Reliability,safetyinformally
IntermedDevelop-ment
MoreFormalTest ofcomponents
MoreMatureSystem
Tests in Actual use;Externalreviewers
LargeClinicaltrial,? RCT
WiderImplemen-tation
Post-Fieldingsurveillance
After Miller RA JAMIA 1996
Chan AS et al Post Fielding Surveillance... Advances in Patient Safety: From Research to Implementation. Vol. 1. Research Findings AHRQ Publication Number 05-0021-1