Eliz Markowitz 1,4, MS Committee: Jiajie Zhang 1,3, PhD (Chair) Elmer V. Bernstam 1,3, MS, MSE, MD...

Eliz Markowitz 1,4, MS Committee: Jiajie Zhang 1,3, PhD (Chair) Elmer V. Bernstam 1,3, MS, MSE, MD Jorge Herskovic 3,4, MD, PhD Todd R. Johnson 2,4, PhD 1 School of Biomedical Informatics, The University of Texas Health Science Center at Houston, TX; 2 Division of Biomedical Informatics, Department of Biostatistics, University of Kentucky, Lexington, KY; 3 The University of Texas MD Anderson Cancer Center Houston, Texas 4 National Center for Cognitive Informatics and Decision Making in Healthcare 1

Problem Statement Background and Significance Preliminary Studies Specific Aims Timeline 2

Systematic, consistent approaches Can improve Efficiency Safety Effectiveness Examples Standard operating procedures Clinical Guidelines Decision support Hard stops in EHRs But flexibility is needed to accommodate variation common to healthcare 3

4 House staff reported that because of CPOE inflexibility, nonstandard specificationsare often impossible to enter.

5 systems optimized to acquire structured data from healthcare providers often have user interfaces that are idiosyncratic, inflexible, or inefficient, and thus place the burden of entering the data in a structured format on a busy healthcare provider, rather than leveraging specific computer programs to extract the data from the human-input clinical narrative.

Too much systematicity Decreases quality and efficiency Lead to caregiver resistance Creative Workarounds Diabeetes Too much flexibility Decreases quality and efficiency How do we find the right blend? 6

A set of parts working together as parts of a mechanism or an interconnecting network. An integrated set of elements that accomplish a defined objective. 9

Decades of work in industrial systems engineering, manufacturing, business process modeling Flexibility No precise definition The amount of variation that can be tolerated while maintaining operation in the desired range The relevant dimensions depend on the process Trade-offs 10

Perer & Shneidermans Seven SYF Design Goals See an overview of the sequential process of actions Step through actions Select actions in any order See completed and remaining actions Annotate their actions Share progress with other users Reapply past paths of exploration on new data 11

Systematicity Provides structure to ensure consistency, efficiency, and safety by imposing necessary structure Flexibility The ability to constantly adapt to circumstances and still reach the goal state 12

Guide the design of systems that support graceful degradation from idealized practices to those that are more suitable for a given situation Allow exploration of trade-offs among designs Example: Flexibility vs. Interface Complexity Provide an objective measure of flexibility for comparing designs 13

Identify a task ( a problem to be solved) Analyze three problem spaces Idealized space: The best or idealized practice Natural space: Natural constraints on task performance System space: The new or redesigned system Qualitatively evaluate and compare each space Quantitatively compare flexibility measures for each space 16

Idealized: The best practice Natural: All possible practices in the real world System: How the work is done in the (re)designed system 17

Central Line (CL) Insertion Used to deliver medications and/or fluids Necessary Actions Sterilize Site Drape patient Put Hat On Put Mask On Put Gown On Wash hands Glove up Insert Central Line Apply Sterile Dressing Central Line Insertion 18

Describe how the existing system should be used to carry out the task Work Domain Ontology (Zhang et. al): Defines the explicit, abstract, implementation- independent parts of a task Focus on essential functionality Sets the scope of the model Specifies the necessary assumptions 19

Specified as a WDO for the task. The assumptions allow us to better assess the validity and scope of the idealized space Assumptions A single caregiver accomplishes the entire task All supplies needed to do the task are available There is sufficient time to accomplish the procedure according to best practices 20

13,004 paths to any state in which the central line is inserted 1,680 possible paths to the idealized state Only 2 paths contain the appropriate sequence of 9 actions The natural central line insertion space 21

Alter the operators and conditions of the natural and ideal space Based upon CL Insertion intervention proposed by Berenholtz et. al. Has lowered incidence of bloodstream infections to nearly zero Encourages adherence to best practices 22

Berenholtz CL Insertion intervention Educate staff CL insertion cart Ask daily if CL can be removed Checklist Nurse empowered to halt procedure if guidelines are not followed in a non- emergent situation 23

Amount of information measured in bits The amount of information between n equally likely actions is log 2 (n) Amount of information in a choice taken with probability p i is log 2 (1/p i ) FLEXIBILITY MEASURES Thus, the average information over all paths where each path has probability pi is:: Can convert to percentage flexibility: 24 SHANNON INFORMATION

Idealized Two equivalent paths of 9 actions Each has probability.5 2*.5*Log 2 (1/.5) = 1 Only requires 1 bit of information 10% flexibility 25

26 Shortest Path 1 step Average 9.62 bits per path Total bits in either correct path is 15.2992 64.96% flexibility The natural central line insertion space

Difference from natural space due to the initial decision; emergent or non-emergent situation Shortest Path 1 step Average 6.31 bits per path 43.8% flexibility 27

Flexibility of CL Insertion spaces using average bits per path and % flexibility measures 28

A perfect user must perform 9 steps to correctly perform the task A random user needs an average of 147 steps before performing the task correctly Includes restarting if task is not completed correctly In reality, this means a random user would do the task incorrectly much more often than correctly 29

As user knowledge increases, user performance nears that of the optimal performance Users Perception Training Adding Constraints Alter System Design 30 Knowledge Steps

CDSS-CPOE systems have been shown to decrease medical errors, improve efficiency, and enhance clinical performance. However, CPOE systems can also cause adverse consequences. We hypothesize that someof the unintended consequences are due to a mismatch between system and task flexibility. 32

All Paths 7 steps Total Paths 12 Average 3.5849 bits per path 33.8684% flexibility

Shortest Path 9 Total Paths 5040 Average 10.6907 bits per path 65.886% flexibility NATURAL SPACE

36 Ingredient Form Dose Frequency Duration Refills

All Paths 8 Total Paths 2 Average 1 bits per path 11.11% flexibility

Captures intuitive notion of flexibility Can work with spaces that have cycles Average bits per task captures performance across the range of user experience 38

The long-term goal of this work is to guide the design of systems that allow graceful degradation from ideal performance on standard cases to more flexible performance for coping with variations. We hypothesize that optimal performance is reached when task flexibility matches interface flexibility. 41

Evaluate the flexibility-compatibility hypothesis. We hypothesize that user performance will improve when the flexibility of the system interface matches the flexibility of the task. 42

Design & Implement Medication Entry Interfaces High Interface Flexibility (HIF) Low Interface Flexibility (LIF) Controlled Study Analyze 43

TD170.304.a 1.1: Medication Orders Amoxil 250 mg oral suspension, Disp #150 ml, Sig: Take 5 ml q8h X 10 days, 0 Refills Plavix 75 mg tablet, Disp #30, Sig: Take 1 tablet QD, 1 Refill Catapres 0.1 mg tablet, Disp #60, Sig: Take 1 tablet bid, 1 Refill TD170.304.a 1.4: Medication Orders Glyburide (Diabeta) 2.5 mg tablet, Disp # 60, Sig: Take 1 tablet PO, Q AM, 0 Refills Atorvastatin calcium (Lipitor) 10 mg tablet, Disp # 60, Sig: Take 1 tablet PO Qday, 1 Refill Candesartan cilexetil (Atacand), 16 mg tablet, Disp# 60, Sig: Take 1 tablet PO QDay, 1 Refill 44

Medication Orders Ambien CR 12.5 mg tablet, Disp #30, 1 Refill Divalproex 250 mg tablet, Disp #60, Sig: Take 2 tablet PO bid Doxycycline 100 mg, Disp #14, 0 Refills Medication Orders Clyndamycin 300 mg tablet, Disp # 28, 0 Refills Viagra 50 mg tablet, Disp # 30 Vicodin, 7.5 mg tablet, Disp# 60, Sig: Take 2 tablet PO bid 45

Test FCH in 2 x 2 Factorial 46

99% Confidence Power of 0.97 13 individuals/sample 26 Total 47

HIF will perform best in the scenario in which both the task and system are highly flexible. LIF will perform best in the scenario in which both the task and system are highly inflexible. Both interfaces will perform adequately in the remaining scenarios. 48

Evaluate the prediction that soft constraints, which suggest a systematic approach to a task, can provide equal or better overall performance than hard constraints. 49

One Size Fits All is not feasible Problem of excessive rigidity Hard constraints Problem of excessive flexibility No constraints or soft constraints 50

As user knowledge increases, user performance nears that of the optimal performance Automating parts of a task, via either soft or hard constraints, can allow the user to start closer to the optimal number of steps. 51 Knowledge Steps Knowledge Steps

Analyze user performance User error rates, completion rates, and completion time Gauge user satisfaction IBM CUSQ: Computer Usability Satisfaction Questionnaire 52

Test effectiveness of Soft and Hard constraints in 2 x 2 design 53

IBM Computer Usability Satisfaction Questionnaire Gauges User Interface Satisfaction 1-7 Scale Overall User Reaction 54

Average task performance is better, measured by user error rates and task completion time, when using an interface with soft constraints than an interface with hard constraints. User satisfaction will be greater for interfaces with soft constraints than those with hard constraints. 56

Evaluate SYFSA predictions that systems that offer less support for the constraints in the idealized task space will lead to longer novice task times and more trials to reach expert performance. 57

SYFSA and Hick-Hyman The time to make a decision is proportional to the amount of information in the available choices High flexibility leads to more time to become an expert user Constraints aid sequence memorization 58

Knowledge/Usability Analysis on HIF and LIF Controlled trial Users test out two interfaces User completion time, attempts and total time taken to reach criterion, user error rates User Satisfaction 59

As user knowledge increases, user performance nears that of the optimal performance Users perception of difficulty decreases as knowledge increases 60 Knowledge Steps Knowledge Steps

User performance declines when using an interface having fewer constraints than identified appropriate by the idealized space. The rate of learnability will be lower for interfaces that offer less support for the constraints than determined by the idealized space. User satisfaction will be greater for interfaces in which the constraints of the idealized space match the constraints of the interface. 62

Ensuring flexibility is the sole variable in the experiment Create interfaces rather than use COTS software Subject Recruitment Reduce the power to less than 0.80 Alter the experiment from a between-groups to a within-groups analysis Recruit subjects outside of SBMI 63

Arc of 3 papers to be completed by January 2014. 65

My super awesome committee Grant No. 10510592 for Patient-Centered Cognitive Support under the Strategic Health IT Advanced Research Projects (SHARP) from the Office of the National Coordinator for Health Information Technology. A pre-doctoral training fellowship from the Agency for Healthcare Research and Quality 66

Ask em! 67

REFERENCES [1] Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform. 2005;38(1):75-87. [2] Keohane CA, Bates DW. Medication safety. Obstet Gynelo Clin North Am. 2008 Mar;35(1):37-52. [3] Roughead EE, Semple SJ. Medication safety in acute care in Australia: where are we now? Part 1: a review of the extend and causes of medication problems. Aust New Zealand Health Policy. 2009;6:18. [4] De Koning JS, Klazinga NS, Koudstaal PJ, Prins A, Dippel DW, Heerings J, et al. Quality of care in stroke prevention: results of an audit study among general practitioners. Prev Med.2004;38:129-36. [5] Poon EG. Medication reconciliation: whose job is it? AORN J. 2009 Jun;89(6):1180, 1122. [6] Perer A, Shneiderman B. Systematic yet flexible discovery: guiding domain experts through exploratory data analysis. Proceedings of the 13 th International Conference on Intelligent User Interfaces; 2008; New York, NY. 109-118. [7] Goel V, Pirolli P. The structure of design problem spaces. Cognitive Science. 1992;16:395-429. [8] Card SK, Moran TP, Newell A. The psychology of human-computer interaction. Hillsdale, NJ: Erlbaum. 1983. [9] Newell A, Simon HA. Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall. 1992. [10] Cross N. Developments in design methodology. New York: Wiley. 1984. [11] Zhang J, Butler K. UFuRT: A work-centered framework and process for design and evaluation of information systems. Proceedings of HCI International. 2007. [12] Swaney RE and Grossmann IE. An index for operational flexibility in chemical process design. Part I: Formulation and theory. AIChE Journal 31, 4 (1985), 621630. [13] Sethi A. and Sethi S. Flexibility in manufacturing: A survey. International Journal of Flexible Manufacturing Systems 2, 4 (1990). [14] Vokurka RJ and OLeary-Kelly SW. A review of empirical research on manufacturing flexibility. Journal of Operations Management 18, 4 (2000), 485501. [15] Bider, I. State-oriented business process modeling: principles, theory and practice. Royal Institute of Technology and Stockholm University, (2002). [16] Gebauer, J. and Schober, F. Information System Flexibility and the Performance of Business Processes. Working Papers, (2005). [17] Shi, D. and Daniels, R.L. A survey of manufacturing flexibility: implications for e-business flexibility. IBM Systems Journal 42, 3 (2003), 414427. [18] Soffer P. On the notion of flexibility in business processes. Proceedings of the CAiSE, (2005), 3542. [19] Regev G, Bider I, and Wegmann A. Defining business process flexibility with the help of invariants. Software Process: Improvement and Practice 12, 1 (2007), 6579. [20] Wickens CD, Carswell MC. The proximity compatibility principle: its psychological foundation and relevance to display design. Human Factors. 1995;37. [21] Carswell MC, Wickens CD. Mixing and matching lower-level codes for object displays: evidence for two sources of proximity compatibility. Human Factors. 1996. 38. [22] Wickens CD, Hollands JG. Engineering Psychology and Human Performance. 3 rd ed. Upper Saddle River, NJ : Prentice Hall; 2000. [23] Shneiderman B. Creativity support tools: accelerating discovery and innovation. Communications of the ACM. 2007 Dec;50(12):20-32. [24] Berenholtz, S.M., Pronovost, P.J., Lipsett, P.A., et al. Eliminating catheter-related bloodstream infections in the intensive care unit. Critical Care Medicine 32, 10 (2004), 2014-2020. [25] Shannon CE and Weaver W. The Mathematical Theory of Communication. University of Illinois Press, 1963. [26] Hick WE. On the rate of gain of information. The Quarterly Journal of Experimental Psychology 4. 1952;1:11-26. [27] Hyman R. Stimulus information as a determinant of reaction time. Hournal of Experimental Psychology 45. 1953;3:188-196. [28] Pronovost P, Needham D, Berenholtz S, et. al. An intervention to decrease cather-related bloodstream infections in the ICU. New England Journal of Medicine. 2006;355(26):2725-32. [29] Leape LL, Bates DW, Cullen DJ, et al: Systems analysis of adverse drug events. ADE Prevention Study Group. JAMA 1995; 274: 3543 [30] Schneider PJA: Review of the safety of intravenous drug delivery systems. Hospital Pharmacy. 1999; 34:10441056 [31] Tissot E, Cornette C, Demoly P, et al: Medication errors at the administration stage in an intensive care unit. Intensive Care Medicine. 1999; 25:353359 [32] van den Bemt PM, Fijn R, van der Voort PH, et al: Frequency and determinants of drug administration errors in the intensive care unit. Crit Care Med 2002; 30:846850 [33] Herout P, Erstad B: Medication errors involving continuously infused medications in a surgical intensive care unit. Crit Care Med. 2004; 32:428432 [34] Cummings K and McGowan R. Smart infusion pumps are selectively intelligent. Nursing. 2011; March: 59 [35] Proceedings From the ISMP Summit on the Use of Smart Infusion Pumps: Guidelines for Safe Implementation and Use. Philadelphia, PA: Institute for Safe Medication Practices; 2009. [36]Neilsen J, Molich R. Heuristic evaluation of user interfaces. Proceedings of CHI90; 1990 April 1-5; Seattle, WA. 249-256, 1990. [37]Nielsen J. Severity ratings for usability problems. Use-it.com. 2005. Available at: http://www.useit.com/papers/heuristic/severityrating.html. Accessed March 1, 2010. [38] Ellitt GR, Engblom E, Aslani P, Westerlund T, Chen TF. Drug related problems after discharge from an Australian teaching hospital. Pharm World Sci. 2010. [39] Joint Commission on Accreditation of Healthcare Oganizations. Using medication reconciliation to prevent errors. Sentinel Event Alert. January 25, 2006;(35):1-4. [40] Johnson CM, Marcy TR, Harrison DL, Young RE, Stevens EL, Shadid J. Medication reconciliation in a community pharmacy setting. J Am Pharm Assoc. 2010 Jul-Aug;50(4):523-6. [41] John BE, Kieras DE. Using GOMS for user interface design and evaluation: which technique? ACM Transactions of Computer-Human Interaction (TOCHI). 1996;3(4):287-319. [42] Card SK, Moran TP, & Newell A. The keystroke-level model for user performance time with interactive systems. Communications of the ACM. 1980; 23(7), 396-410. [43] Kieras D. Using the keystroke-level model to estimate execution times. Unpublished report. Ann Arbor: Department of Psychology, University of Michigan; 2001. 68

Eliz Markowitz 1,4, MS Committee: Jiajie Zhang 1,3, PhD (Chair) Elmer V. Bernstam 1,3, MS, MSE, MD...

Documents

Transcript of Eliz Markowitz 1,4, MS Committee: Jiajie Zhang 1,3, PhD (Chair) Elmer V. Bernstam 1,3, MS, MSE, MD...