MDVita - Drummond Group LLC...Electronic Health Record Usability Testing (EHRUT) Report of MDVita...

Electronic Health Record Usability Testing (EHRUT) Report of MDVita version 25

Health Care 2000, Inc. 7205 Corporate Drive, Suite 401, Miami, FL 33126. Tel (305) 262-5959 of 66

Electronic Health Record Usability Testing Report

MDVita version 25



EHR Usability Test Report of MDVita version 25

Report based on ISO/IEC 25062:2006(E) Common Industry Format for Usability Test Reports

MDVita EHR version 25

Date of Usability Test: 12/13/2017 – 12/21/2017

Date of Report: 12/26/2017. Last Modified: 01/02/2018

Report Prepared By: HealthCare 2000, Inc

Ileana Pichel, System Analyst

(305) 262-5959 ext. 0

[email protected]

7205 Corporate Center Drive, Suite 401, Miami, FL 33126

Table of Contents 1. EXECUTIVE SUMMARY ........................................................................................................................ 4

2. INTRODUCTION .................................................................................................................................... 8

3. METHOD ................................................................................................................................................ 8

3.1. PARTICIPANTS .............................................................................................................................. 8

3.2. STUDY DESIGN ............................................................................................................................. 9

3.3. TASKS .......................................................................................................................................... 10

3.4. PROCEDURES............................................................................................................................. 11

3.5. TEST ENVIRONMENT ................................................................................................................. 12

3.6. TEST FORMS AND TOOLS ......................................................................................................... 12

3.7. PARTICIPANT INSTRUCTIONS .................................................................................................. 12

3.8. USABILITY METRICS .................................................................................................................. 13

3.9. DATA SCORING........................................................................................................................... 13

4. RESULTS ............................................................................................................................................. 15

4.1. DATA ANALYSIS AND REPORTING .......................................................................................... 15

4.2. DISCUSSION OF THE FINDINGS ............................................................................................... 18

4.2.1. EFFECTIVENESS ................................................................................................................ 18

4.2.2. EFFICIENCY ......................................................................................................................... 18

4.2.3. SATISFACTION .................................................................................................................... 18

4.2.4. MAJOR FINDINGS ............................................................................................................... 18

4.2.5. AREAS FOR IMPROVEMENT ............................................................................................. 19

5. APPENDICES ....................................................................................................................................... 19

5.1. Appendix 1: SAMPLE RECRUITING SCREENER....................................................................... 21

5.2. Appendix 2: PARTICIPANT DEMOGRAPHICS ........................................................................... 23

5.3. Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ............. 24



5.4. Appendix 4: MODERATOR’S GUIDE .......................................................................................... 26

5.4.1. EHRUT Usability Test ............................................................................................................. 26

5.5. Appendix 5: Tasks Sheet .............................................................................................................. 28

5.6. Appendix 6: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................... 65



EXECUTIVE SUMMARY

A usability test of MDVita v 25 was conducted between December 13th, 2017 and December 21st, 2017

by Health Care 2000, Inc in one-to-one sessions with the test participants. The purpose of this test was to

test and validate the usability of the current user interface, and provide evidence of usability in the EHR

Under Test (EHRUT). During the usability test, 10 healthcare providers matching the target demographic

criteria served as participants and used the EHRUT in simulated, but representative tasks. The participants

are representative of our user base.

This study collected performance data on ten tasks typically conducted on an EHR:

170.315(a)(1) CPOE – Medications

170.315(a)(2) CPOE – Laboratory

170.315(a)(3) CPOE – Diagnostic Imaging

170.315(a)(4) Drug-drug, Drug-allergy Interaction Checks

170.315(a)(5) Demographics

170.315(a)(6) Problem List

170.315(a)(7) Medication List

170.315(a)(8) Medication Allergy List

170.315(a)(9) Clinical Decision Support

170.315(a)(14) Implantable Device List

170.315(b)(2) Clinical Information Reconciliation and Incorporation

During the three hours one-on-one usability test, each participant was greeted by the administrator and

asked to review and sign an informed consent/release form (included in Appendix 3); they were instructed

that they could withdraw at any time. Participants had prior experience with the EHR. The administrator

introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the

EHRUT. During the testing, the administrator timed the test and, along with the data logger(s) recorded

user performance data on paper and electronically. The administrator did not give the participant assistance

in how to complete the task.

The Customized Common Industry Format Template for Electronic Health Record Usability Testing (NISTIR

7742) (http://ws680.nist.gov/publication/get_pdf.cfm?pub_id=907312) is used for usability test reports

(ISO/IEC 25062:2006(E)).

The following types of data were collected for each participant:

Number of tasks successfully completed within the allotted time without assistance

Time to complete the tasks

Number and types of errors

Path deviations

Participant’s verbalizations

Participant’s satisfaction ratings of the system



All participant data was de-identified – no cross reference could be made from the identity of the participant

to the data collected. Following the conclusion of the testing, participants were asked to complete a posttest

questionnaire. Participants were not compensated for their time. Various recommended metrics, in

accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the

Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a

summary of the performance and rating data collected on the EHRUT.

Task

Nbr

Measure Task Success

(%)

(Success/Failure)

Path Deviation

Mean (SD)

Task Time

Deviations (Observed / Optimal)

Errors

Mean (SD)

Task Ratings 5=Easy

Mean (SD)

1 Create Medication Order 100/0 10(0) 74/90 0(0) 5(0)

2 Change Medication Order 100/0 10(0) 71/90 0(0) 5(0)

3 Access Medication Order 100/0 4(0) 11/15 0(0) 5(0)

4 Create Laboratory Order 100/0 11(0) 35/45 0(0) 5(0)

5 Change Laboratory Order 100/0 9(0) 34/45 0(0) 5(0)

6 Access Laboratory Order 100/0 5(0) 14/20 0(0) 5(0)

7 Create Imaging (Radiology) Order

100/0 13(0) 45/60 0(0) 5(0)

8 Change Imaging (Radiology) Order

100/0 9(0) 20/25 0(0) 5(0)

9 Access Imaging (Radiology) Order

100/0 7(0) 13/20 0(0) 5(0)

10 Adjust the severity level for drug-drug notifications

100/0 10(9) 19/30 0(0) 5(0)

11 Act upon drug-drug and drug-allergy interventions

100/0 12.8(0.4) 70/90 0.2(0.4) 5(0)

12 Record patient’s Demographics 100/0 11.8(0.4) 79/90 0.2(0.4) 5(0)

13 Change patient’s Demographics 100/0 8(0) 37/45 0(0) 5(0)

14 Access patient’s Demographics 100/0 4(0) 17/30 0(0) 5(0)

15 Record Problem in Problem List 100/0 7(0) 23/30 0(0) 5(0)



16 Change Problem List 100/0 7(0) 22/30 0(0) 5(0)

17 Access Problem List 100/0 5(0) 14/20 0(0) 5(0)

18 Record medication in Medication List

100/0 8(0) 20/30 0(0) 5(0)

19 Change Medication List 100/0 8(0) 16/20 0(0) 5(0)

20 Access Medication List 100/0 5(0) 11/15 0(0) 5(0)

21 Record Medication Allergy 100/0 10(0) 24/30 0(0) 5(0)

22 Change Medication Allergy 100/0 6(0) 15/20 0(0) 5(0)

23 Access Patient’s Allergy List 100/0 4(0) 10/15 0(0) 5(0)

24 Configure CDS Interventions 100/0 9(0) 28/30 0(0) 5(0)

25 Act upon Problem List interventions and Access evidence-based, diagnostic and therapeutic reference resources

100/0 12(0) 27/30 0(0) 5(0)

26 Act upon Medication List interventions and Access evidence-based, diagnostic and therapeutic reference resources

100/0 9(0) 24/30 0(0) 5(0)

27 Act upon Medication Allergy List interventions

100/0 11(0) 29/35 0(0) 5(0)

28 Record patient’s implantable device

100/0 8(0) 47/60 0(0) 5(0)

29 Change patient’s implantable device

100/0 4(0) 15/20 0(0) 5(0)

30 Access patient’s implantable device List

100/0 4(0) 13/20 0(0) 5(0)

31 Reconcile patient medication, problem and medication allergy lists with CCDA and generate resulting CCDA

100/0 27(0) 93/120 0(0) 5(0)

32 CDS. Act upon Demographics intervention. Access evidence-based, diagnostic and therapeutic reference resources and review bibliographic citation, developer, funding source, and release/revision date

100/0 9(0) 50/45 0(0) 5(0)

33 CDS. Act upon Laboratory test result intervention

100/0 8(0) 48/50 0(0) 5(0)



34 CDS. Act upon Vital signs intervention

100/0 8(0) 33/35 0(0) 5(0)

35 CDS. Act upon combination (Demographics and Medication) intervention

100/0 10(0) 29/30 0(0) 5(0)

36 CDS. Act upon CCDA reconciliation intervention

100/0 26(0) 127/130 0(0) 5(0)

The results from the System Usability Scale scored the subjective satisfaction with the system based on

performance with these tasks to be: 100.1

In addition to the performance data, the following qualitative observations were made:

- Major findings

All of the tasks were found easy to use by all users. This is due to improvement made to the UI in the

evaluated version 25 vs. the previous attested version 20, based on user feedbacks.

Some users had difficulty finding the tabs and fields to record and complete the whole patient demographics

task. Some users complained that too much information was needed and found the UI data elements too

far apart.

Some users verbally praised how easy was and intuitive to use the reconciliation tasks.

Some users had experience with other EHRs and praised MDVita for being so well-organized and how

easy and complete it is.

- Areas for improvement

Considering the information gathered from the users, the following areas of functionality have been

identified as in need of enhanced usability design:

Record and Change Patient’s Demographics

Demographics MU core components must be grouped close to each other in the UI for the users to identify

and enter faster.

1 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan

Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores

over 80 would be considered above average.



INTRODUCTION

The EHRUT tested for this study was MDVita version 25. Designed to present medical information to

healthcare providers in General and Primary Care practices, the EHRUT consists of health care practice

management, electronic health records, clinical notes, orders, patient chart with clinical information,

ePrescribe, and treatment plans. The usability testing attempted to represent realistic exercises and

conditions.

The purpose of this study was to test and validate the usability of the current user interface, and provide

evidence of usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency

and user satisfaction, such as time taken to complete task, and the user’s feedbacks, were captured during

the usability testing.

METHOD PARTICIPANTS

A total of 10 participants, that were representative of our user base, were tested on the EHRUT(s).

Participants in the test were one physician, one registered nurse, seven medical assistances and one office

supervisor. Participants were recruited by HealthCare 2000, Inc from its client base and were not

compensated for their time. In addition, participants had no direct connection to the development of or

organization producing the EHRUT(s). Participants were not from the testing or supplier organization.

Participants were given the opportunity to have the same orientation and level of training as the actual end

users would have received.

For the test purposes, end-user characteristics were identified and translated into a recruitment screener

used to solicit potential participants; an example of a screener is provided in Appendix 1.

Recruited participants had a mix of backgrounds and demographic characteristics conforming to the

recruitment screener. The following is a table of participants by characteristics, including demographics,

professional experience, computing experience and user needs for assistive technology. Participant names

were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.



Part ID Gender Age Education Occupation/ role Professional

Experience Computer

savvy

Previous

EHR

Experience

Assistive Technology

Needs

1 User 1 F 40-59 High

School

Medical

Assistance

7 years Yes Yes No

2 User 2 F 40-59 College Medical

Assistance

5 years Yes Yes No

3 User 3 M 23-39 Post

Graduate

Medical

Assistance

3 years Yes Yes No

4 User 4 F 23-39 High

School

Medical

Assistant

7 years Yes Yes No

5 User 5 F 40-59 College Registered

Nurse

10 years Yes Yes No

6 User 6 M 40-59 Vocational

Training

Medical

Assistant

8 years Yes Yes No

7 User 7 F 23-39 Associate

degree

Medical

Assistant

7 years Yes Yes No

8 User 8 F 23-39 Vocational

Training

Office

Supervisor

3 years Yes Yes No

9 User 9 M 23-39 Doctorate

Degree,

MD

Medical

Assistant

1 year Yes Yes No

10 User

10

M 40-59 Doctorate

Degree,

MD

Physician 3 years Yes Yes No

All ten participants (matching the demographics in the section on Participants) were recruited and all ten

participated in the usability test. A document was sent to the participants to get their demographic

characteristics as shown in Appendix 1.

STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well – that is,

effectively, efficiently, and with satisfaction – and areas where the application failed to meet the needs of

the participants. The data from this test may serve as a baseline for future tests with an updated version

of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this

testing serves as both a means to record or benchmark current usability, but also to identify areas where

improvements must be made.

During the usability test, participants interacted with EHRUT. Each participant used the system via a Laptop

computer, and was provided with the same instructions. The system was evaluated for effectiveness,

efficiency and satisfaction as defined by measures collected and analyzed for each participant:

Number of tasks successfully completed within the allotted time without assistance

Time to complete the tasks

Number and types of errors



Path deviations

Participant’s verbalizations (comments)

Participant’s satisfaction ratings of the system

Additional information about the various measures can be found in Section 3.8 on Usability Metrics.

TASKS

A number of tasks were constructed that would be realistic and representative of the kinds of activities a

user might do with this EHR and were tested in the order of forced-ranked risk-based severity by our end

users, including:

1. 170.315 (a)(1) Computerized Provider Order Entry (CPOE) - Medications

Create Medication Order

Change Medication Order

Access Medication Order

2. 170.315 (a)(2) CPOE - Laboratory

Create Laboratory Order

Change Laboratory Order

Access Laboratory Order

3. 170.315 (a)(3) CPOE - Imaging

Create Imaging (Radiology) Order

Change Imaging (Radiology) Order

Access Imaging (Radiology) Order

4. 170.315 (a)(4) Drug-Drug, Drug Allergy Interaction Checks

Adjust the severity level for drug-drug notifications

Act upon drug-drug and drug-allergy interventions prior medication order completion

5. 170.315 (a)(5) Demographics

Create a Patient entering Date of Birth, Birth Sex, Preferred Language, Race, Ethnicity, Sexual

Orientation and Gender identity

Change demographics

Access Patient’s Demographics

6. 170.315 (a)(6) Problem List

Record a Patient problem using SNOMED

Change Problem Status

Access Problem List

7. 170.315 (a)(7) Medication List

Record medication in the Medication List

Change Medication List

Access Medication List



8. 170.315 (a)(8) Medication Allergy List

Record medication allergy

Change medication allergy

Access Patient’s Allergy List

9. 170.315 (a)(9) Clinical Decision Support (CDS)

Configure CDS interventions

Act upon Problem List interventions

Act upon Medication List interventions

Act upon Medication Allergy List interventions

Act upon Demographic interventions

Act upon Laboratory test result intervention

Act upon Vital Signs interventions

Act upon a combination of at least two elements above

Access evidence-based, diagnostic and therapeutic reference resources for Problem List,

Medication List, and Demographics interventions

10. 170.315 (a)(14) Implantable Device List

Record an implantable device

Change an implantable device

Access Patient’s implantable device List

11. 170.315 (b)(2) Clinical Information Reconciliation and Incorporation

Reconcile patient medication list with a CCDA

Reconcile patient problem list with a CCDA

Reconcile patient medication allergy list with a CCDA

Tasks were selected based on their relation to the 2015 Edition Meaningful Use criteria, criticality of

function, and those that may be most troublesome for users.

PROCEDURES

Participants were met at their location as coordinated beforehand in the schedule; their identity was verified

and matched with a name on the participant schedule. Participants were then assigned a participant ID.

Each participant reviewed and signed an informed consent and release form (See Appendix 3) sent by

administrator beforehand.

The usability testing administrator conducting the test was a System Analyst who is also a Registered Nurse

with 10 years of experience in healthcare.

The administrator moderated the session including administering instructions and tasks. The administrator

also monitored task times, obtained post-task rating data, and took notes on participant comments. The

administrator also served as the data logger and took notes on task success, path deviations, number and

type of errors, and comments.

Participants were instructed to perform the tasks (see specific instructions below):



As quickly as possible making as few errors and deviations as possible.

Without assistance; administrators were allowed to give immaterial guidance and clarification on

tasks, but not instructions on use.

Without using a think aloud technique.

For each task, the participants were given a written copy of the task a day before the session took place.

Task timing began once the administrator finished reading the question. The task time was stopped once

the participant indicated they had successfully completed the task. Scoring is discussed below in Section

3.9.

Following the session, the administrator gave the participant the post-test questionnaire (e.g., the System

Usability Scale, see Appendix 5), and thanked each individual for their participation.

Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses,

and post-test questionnaire were recorded into a form document.

TEST ENVIRONMENT

The test was performed in a one-on-one session at the participant’s location. The participants started the

session using a Laptop computer provided by the administrator.

TEST FORMS AND TOOLS

During the usability test, various documents and instruments were used, including:

1. Informed Consent

2. Moderator’s Guide

3. Post-test Questionnaire

Examples of these documents can be found in Appendices 3-5 respectively. The Moderator’s Guide was

devised so as to be able to capture required data.

PARTICIPANT INSTRUCTIONS

The administrator reads the following instructions aloud to the each participant (also see the full moderator’s

guide in Appendix 4):

Thank you for participating in this study. Your input is very important. Our session today will last about 3

hours. During that time you will use an instance of an electronic health record. I will ask you to complete a

few tasks using this system and answer some questions. You should complete the tasks as quickly as

possible making as few errors as possible. Please try to complete the tasks on your own following the

instructions very closely. Please note that we are not testing you we are testing the system, therefore if

you have difficulty all this means is that something needs to be improved in the system. I will be here in

case you need specific help, but I am not able to instruct you or provide help in how to use the application.



Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to

you, and how we could improve it. I did not have any involvement in its creation, so please be honest with

your opinions. All of the information that you provide will be kept confidential and your name will not be

associated with your comments at any time. Should you feel it necessary you are able to withdraw at any

time during the testing.

Following the procedural instructions, participants were shown the EHR and as their first task, were given

time (30 minutes) to explore the system and make comments. Once this task was complete, the

administrator gave the following instructions:

For each task, I will read the description to you and say “Begin.” At that point, please perform the task and

say “Done” once you believe you have successfully completed the task. I would like to request that you not

talk aloud or verbalize while you are doing the tasks. I will ask you your impressions about the task once

you are done.

Participants were then given 31 tasks to complete.

USABILITY METRICS

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health

Records, EHRs should support a process that provides a high level of usability for all users. The goal is for

users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this

end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing.

The goals of the test were to assess:

1. Effectiveness of EHRUT by measuring participant success rates and errors

2. Efficiency of EHRUT by measuring the average task time and path deviations

3. Satisfaction with EHRUT by measuring ease of use ratings

DATA SCORING The following table (Table 3) details how tasks were scored, errors evaluated, and the time data analyzed.



Measures Rationale and Scoring

Effectiveness:

Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis.

The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.

Task times were recorded for successes. Observed task times divided

by the optimal time for each task is a measure of optimal efficiency.

Optimal task performance time, as benchmarked by expert

performance under realistic conditions, is recorded when constructing

tasks. Target task times used for task times in the Moderator’s Guide

must be operationally defined by taking multiple measures of optimal

performance and multiplying by some factor 2.5that allows some time

buffer because the participants are presumably not trained to expert

performance. Thus, if expert, optimal performance on a task was [20

seconds then allotted task time performance was [20 * 1.25] 50

seconds. This ratio should be aggregated across tasks and reported

with mean and variance scores.

Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a “Failures.” No task times were taken for errors.

The total number of errors was calculated for each task and then

divided by the total number of times that task was attempted. Not all

deviations would be counted as errors. This should also be expressed

as the mean number of failed tasks per participant.

On a qualitative level, an enumeration of errors and error types should

be collected.

Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the application was

recorded. Deviations occur if the participant, for example, went to a

wrong screen, clicked on an incorrect menu item, followed an incorrect

link, or interacted incorrectly with an on-screen control. This path was

compared to the optimal path. The number of steps in the observed

path is divided by the number of optimal steps to provide a ratio of path

deviation.

It is strongly recommended that task deviations be reported. Optimal

paths (i.e., procedural steps) should be recorded when constructing

tasks.



Efficiency:

Task Time

Each task was timed from when the administrator said “Begin” until the

participant said, “Done.” If he or she failed to say “Done,” the time was

stopped when the participant stopped performing the task. Only task

times for tasks that were successfully completed were included in the

average task time analysis. Average time per task was calculated for

each task. Variance measures (standard deviation and standard error)

were also calculated.

Satisfaction:

Task Rating

Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. 2

Common convention is that average ratings for systems judged easy to use should be 3.3 or above.

To measure participants’ confidence in and likeability of the EHRUT

overall, the testing team administered the System Usability Scale

(SUS) post-test questionnaire. Questions included, “I think I would like

to use this system frequently,” “I thought the system was easy to use,”

and “I would imagine that most people would learn to use this system

very quickly.” See full System Usability Score questionnaire in

Appendix 5.3

Table 3. Details of how observed data were scored.

RESULTS

DATA ANALYSIS AND REPORTING

The results of the usability test were calculated according to the methods specified in the Usability Metrics

section above.

The usability testing results for the EHRUT are detailed below.

2 See Tedesco and Tullis (2006) for a comparison of post-task ratings for usability tests. Tedesco,

D. & Tullis, T. (2006) A comparison of methods for eliciting post-task subjective ratings in usability

testing. Usability Professionals association Conference, June 12 – 16, Broomfield, CO.

3 The SUS survey yields a single number that represents a composite measure of the overall

perceived usability of the system. SUS scores have a range of 0 to 100 and the score is a relative

benchmark that is used against other iterations of the system.



Task

Nbr

Measure Task Success

(%)

(Success/Failure)

Path Deviation

Mean (SD)

Task Time

Deviations (Observed / Optimal)

Errors

Mean (SD)

Task Ratings 5=Easy

Mean (SD)

1 Create Medication Order 100/0 10(0) 74/90 0(0) 5(0)

2 Change Medication Order 100/0 10(0) 71/90 0(0) 5(0)

3 Access Medication Order 100/0 4(0) 11/15 0(0) 5(0)

4 Create Laboratory Order 100/0 11(0) 35/45 0(0) 5(0)

5 Change Laboratory Order 100/0 9(0) 34/45 0(0) 5(0)

6 Access Laboratory Order 100/0 5(0) 14/20 0(0) 5(0)

7 Create Imaging (Radiology) Order

100/0 13(0) 45/60 0(0) 5(0)

8 Change Imaging (Radiology) Order

100/0 9(0) 20/25 0(0) 5(0)

9 Access Imaging (Radiology) Order

100/0 7(0) 13/20 0(0) 5(0)

10 Adjust the severity level for drug-drug notifications

100/0 10(9) 19/30 0(0) 5(0)

11 Act upon drug-drug and drug-allergy interventions

100/0 12.8(0.4) 70/90 0.2(0.4) 5(0)

12 Record Patient’s Demographics 100/0 11.8(0.4) 79/90 0.2(0.4) 5(0)

13 Change Patient’s Demographics 100/0 8(0) 37/45 0(0) 5(0)

14 Access Patient’s Demographics 100/0 4(0) 17/30 0(0) 5(0)

15 Record Problem in Problem List 100/0 7(0) 23/30 0(0) 5(0)

16 Change Problem List 100/0 7(0) 22/30 0(0) 5(0)

17 Access Problem List 100/0 5(0) 14/20 0(0) 5(0)

18 Record medication in Medication List

100/0 8(0) 20/30 0(0) 5(0)

19 Change Medication List 100/0 8(0) 16/20 0(0) 5(0)



20 Access Medication List 100/0 5(0) 11/15 0(0) 5(0)

21 Record Medication Allergy 100/0 10(0) 24/30 0(0) 5(0)

22 Change Medication Allergy 100/0 6(0) 15/20 0(0) 5(0)

23 Access Patient’s Allergy List 100/0 4(0) 10/15 0(0) 5(0)

24 Configure CDS Interventions 100/0 9(0) 28/30 0(0) 5(0)

25 Act upon Problem List interventions and Access evidence-based, diagnostic and therapeutic reference resources

100/0 12(0) 27/30 0(0) 5(0)

26 Act upon Medication List interventions and Access evidence-based, diagnostic and therapeutic reference resources

100/0 9(0) 24/30 0(0) 5(0)

27 Act upon Medication Allergy List interventions

100/0 11(0) 29/35 0(0) 5(0)

28 Record patient’s implantable device

100/0 8(0) 47/60 0(0) 5(0)

29 Change patient’s implantable device

100/0 4(0) 15/20 0(0) 5(0)

30 Access patient’s implantable device List

100/0 4(0) 13/20 0(0) 5(0)

31 Reconcile patient medication, problem and medication allergy lists with CCDA and generate resulting CCDA

100/0 27(0) 93/120 0(0) 5(0)

32 CDS. Act upon Demographics intervention. Access evidence-based, diagnostic and therapeutic reference resources and review bibliographic citation, developer, funding source, and release/revision date

100/0 9(0) 50/45 0(0) 5(0)

33 CDS. Act upon Laboratory test result intervention

100/0 8(0) 48/50 0(0) 5(0)

34 CDS. Act upon Vital signs intervention

100/0 8(0) 33/35 0(0) 5(0)

35 CDS. Act upon combination (Demographics and Medication) intervention

100/0 10(0) 29/30 0(0) 5(0)

36 CDS. Act upon CCDA reconciliation intervention

100/0 26(0) 127/130 0(0) 5(0)



The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based

on performance with these tasks to be: 100. Broadly interpreted, scores under 60 represent systems with

poor usability; scores over 80 would be considered above average.

DISCUSSION OF THE FINDINGS

All participants rated all of the tasks as ‘Easy’.

All participants completed all the tasks. There were minor deviations to the preferred path while entering

patient demographics, due to errors.

EFFECTIVENESS

Software programs may be functional, the successful performance of the required tasks, the manner in

which a user interacts with the software to perform these tasks may be cumbersome and confusing, causing

it to be unusable. In evaluation of the tested EHR, no tasks failed.

Overall all of the individual tasks tested, tested successfully, resulting in a 100% success rate across all

participants.

EFFICIENCY

Based upon observations of task times and deviations, all of the instances of successful tasks, were

completed within the estimated optimal task performance time. Times were benchmarked by expert

performance under realistic conditions, and multiplied by a 2.5 second time buffer to allow for the fact not

all of the participants in this study are expert users.

SATISFACTION

Participants were asked to rate each task individually on a scale of 1= Very Easy to 5= Very Difficult.

Collectively, all of the tasks were rated with a “1= Very Easy”. The System Usability Scale Questionnaire,

as interpreted referencing Jeff Sauro; Measuring Usability, Quantitative Usability, Statistics & Six Sigma,

indicates this application is user friendly and intuitive in all areas tested.

All participants expressed the application was well organized and was rated easy to learn and easy to use.

MAJOR FINDINGS

Participants were overall very pleased with the product. All of the tasks were found easy to use.



Some users had difficulty finding the tabs and fields to record and complete the whole patient demographics

task. Some users complained that too much information was needed and found the UI data elements too

far apart.

Some users verbally praised how easy was and intuitive to use the reconciliation tasks.

Some users had experience with other EHRs and praised MDVita for being so well-organized and how

easy and complete it is.

AREAS FOR IMPROVEMENT

Considering the information gathered from the users, the following areas of functionality have been

identified as in need of enhanced usability design:

Record and Change Patient’s Demographics

Demographics MU core components must be grouped close to each other in the UI for the users to identify

and enter faster.



APPENDICES

The following appendices include supplemental data for this usability test report. Following is a list of the

appendices provided:

1: Sample recruiting screener

2: Participant demographics

3: Non-Disclosure Agreement (NDA) and Informed Consent Form

4: Example Moderator’s Guide

5: Tasks Sheets

6: System Usability Scale Questionnaire



Appendix 1: SAMPLE RECRUITING SCREENER

Recruiting Script

Hello, my name is Ileana Pichel, calling from HealthCare 2000. We are recruiting individuals to participate in a usability study for an electronic health record. We would like to ask you a few questions to see if you qualify and if would like to participate. This should only take a few minutes of your time. This is strictly for research purposes. Can I ask you a few questions?

1. [If not obvious] Are you male or female?

2. Have you participated in a focus group or usability test in the past 12 months? [If yes, Terminate]

3. Do you, or does anyone in your home, work in marketing research, usability research, web design […etc.]?

[If yes, Terminate]

4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record

software or consulting company? [If yes, Terminate]

5. Would you provide age or age range? _______

6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-

American, Latino/a or Hispanic, etc.]

7. Do you require any assistive technologies to use a computer? [if so, please describe]

Professional Demographics Customize this to reflect your EHR’s primary audience

1. What is your current position and title? (Must be healthcare provider)

RN: Specialty ________________

Physician: Specialty ________________

Resident: Specialty ________________

Administrative Staff

Other ________________________ 2. How long have you held this position?

3. Describe your work location (or affiliation) and environment? (Recruit according to the intended users of

the application) [e.g., private practice, health system, government clinic, etc.] 4. Which of the following describes your highest level of education? [e.g., high school graduate/GED, some

college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)]. Mark below.

• No high school degree

• High school graduate, diploma or the equivalent (for example: GED)

• Some college credit, no degree

• Trade/technical/vocational training

• Associate degree

• Bachelor’s degree

• Master’s degree

• Doctorate degree (e.g., MD, DNP, DMD, PhD)

Computer Expertise Customize this to reflect what you know about your EHR’s audience

1. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR,

research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no

computer use at all, Terminate]

2. About how many hours per week do you spend on the computer? [Recruit according to the demographics

of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]

3. What computer platform do you usually use? [e.g., Mac, Windows, etc.]

4. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]

5. In the last month, how often have you used an electronic health record?



6. How many years have you used an electronic health record?

7. How many EHRs do you use or are you familiar with?

8. MDVita product experience? No ___ Yes ____ how long? ____

9. How does your work environment patient records?

On paper

Some paper, some electronic

All electronic

Contact Information If the person matches your qualifications, ask

Those are all the questions I have for you. Your background matches the people we're looking for.

Would you be able to participate on [date, time]? [If so collect contact information]

May I get your contact information?

Name of participant:

Address:

City, State, Zip:

Daytime phone number:

Evening phone number:

Alternate [cell] phone number:

Email address:



Appendix 2: PARTICIPANT DEMOGRAPHICS

Following is a high-level overview of the participants in this study.

Gender: Men 4 Women 6 Total Participants 10

Occupation/Role:

Physician 1 Medical Assistance 7 Nurse 1 Office Manager 1

Years of Experience:

(0-5) Years 5 (6-10) Years 5



Appendix 3: NON-DISCLOSURE AGREEMENT AND

INFORMED CONSENT FORM

Non-Disclosure Agreement THIS AGREEMENT is entered into as of _ _, 2017, between

_________________________ (“the Participant”) and the testing organization Health Care 2000, Inc.

located at 7205 Corporate Center Drive, Suite 401, Miami, FL 33126.

The Participant acknowledges his or her voluntary participation in today’s usability study may bring the

Participant into possession of Confidential Information. The term "Confidential Information" means all

technical and commercial information of a proprietary or confidential nature which is disclosed by Health

Care 2000, Inc., or otherwise acquired by the Participant, in the course of today’s study.

By way of illustration, but not limitation, Confidential Information includes trade secrets, processes,

formulae, data, know-how, products, designs, drawings, computer aided design files and other computer

files, computer software, ideas, improvements, inventions, training methods and materials, marketing

techniques, plans, strategies, budgets, financial information, or forecasts.

Any information the Participant acquires relating to this product during this study is confidential and

proprietary to Health Care 2000, Inc and is being disclosed solely for the purposes of the Participant’s

participation in today’s usability study.

Participant’s printed name: ___________________________________________

Signature: _____________________________________ Date: ____________________



Informed Consent

Health Care 2000, Inc. would like to thank you for participating in this study. The purpose of this study is to evaluate

an electronic health records system. If you decide to participate, you will be asked to perform several tasks using the

prototype and give your feedback. The study will last about 60 minutes.

Agreement

I understand and agree that as a voluntary participant in the present study conducted by Health Care 2000, Inc. I am

free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study

conducted by Health Care 2000, Inc.

I understand and agree that the purpose of this study is to make software applications more useful and usable in the

future.

I understand and agree that the data collected from this study may be shared with outside of Health Care 2000, Inc

and Health Care 2000’s client. I understand and agree that data confidentiality is assured, because only de-identified

data – i.e., identification numbers not names – will be used in analysis and reporting of the results.

I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can

leave at any time.

Please check one of the following:

YES, I have read the above statement and agree to be a participant. NO, I choose not to

participate in this study.

Signature: __________________________________________ Date: ________________



Appendix 4: MODERATOR’S GUIDE EHRUT Usability Test

Moderator’s Guide

Administrator ________________________

Data Logger ________________________

Date _____________________________ Time _________

Participant # ________

Location ____________________________

Prior to testing

Confirm schedule with Participants

Ensure EHRUT lab environment is running properly

Ensure lab and data recording equipment is running properly

Prior to each participant:

Reset application

Start session recordings with tool

Prior to each task:

Reset application to starting point for next task

Orientation (10 minutes)

Thank you for participating in this study. Our session today will last 3 hours. During that time you will take a look

at an electronic health record system.

I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy

(or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be

asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or

deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot answer help you with

anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the

session as a whole when we can discuss freely.

I did not have any involvement in its creation, so please be honest with your opinions.



The product you will be using today is describe the state of the application, i.e., production version, early prototype,

etc. Some of the data may not make sense as it is placeholder data.

Do you have any questions or concerns?

Preliminary Questions (10 minutes)

What is your job title / appointment?

How long have you been working in this role?

What are some of your main responsibilities?

Tell me about your experience with electronic health records.



Appendix 5: Tasks Sheet

Task 1: Create Medication Order (90 seconds)

Take the participant to the starting point for the task.

While physician is reviewing the Patient’s Medication history, he\she decides to order a Medication for the Patient:

Singulair 10 Mg, take a tablet once a day in the morning. Order 30 units with 6 Refills.

Success:

Easily completed

Completed with difficulty or help :: Describe below

Not completed

Comments:

Task Time: ________ Seconds

Optimal Path:

1. Open Patient Search

2. Search for Patient ‘Jeremy Bates’ and click Select

3. Click “Medication” history

4. Verify Singulair is not one of the active medications

5. Click “Quick Order Medication”

6. Select Patient Insurance, Location (South Dade Medical), Physician, and click Create

7. Click on Diagnosis Tab – Select Diagnosis for medication being ordered (J30.9 – Allergic Rhinitis)

8. Click on Medication Tab - Find Singulair 10 Mg Drug (My Rx – Allergies list) , Click Select

9. Enter SIG: Take 1 Tablet by mouth once a day, in the morning, 30 tablets, 6 refills (Link Med. to DX)

10. Click Save, Acknowledge and Continue, Save and Close

Correct

Minor Deviations / Cycles (Describe below)

Major Deviations (Describe below)

Comments:

Observed Errors and Verbalizations:

Comments:

Rating:

Overall, this task was: _____

Show participant written scale: “Very Difficult” (1) to “Very Easy” (5)

Administrator / Notetaker Comments:



Task 2: Change Medication Order (90 seconds)


After ordering Singulair 10 mg, it is decided to change the medication dose to 5 mg.

Success:

Easily completed


Not completed

Comments:


Optimal Path:




4. Verify Singulair 10mg is on the patient’s medication history

5. Click “Quick Order Medication”

6. Select Patient Insurance, Location (South Dade Medical), Physician, and click Create

7. Click on Diagnosis Tab – Select Diagnosis for medication being ordered (J30.9 – Allergic Rhinitis)

8. Click on Medication Tab - Select Stop on Singulair 10 mg, Save, Acknowledge, Save

9. Under Medication Tab - Find (RX) Singulair 5 mg (My Rx – Allergies list) click select

10. Enter SIG: Take 1 Tablet by mouth once a day, in the morning, 30 tablets, 6 refills (Link Med. to DX)

Save, Acknowledge and Continue, Save and Close

Correct



Comments:


Comments:

Rating:






Task 3: Access Medication Order (15 seconds)


Accessing medication orders of a patient as part of the daily routine.

Success:

Easily completed


Not completed

Comments:


Optimal Path:




4. View medication order history of Singulair drug (+) and Close

Correct



Comments:


Comments:

Rating:






Task 4: Create Laboratory Order (45 seconds)


While reviewing a patient chart, the provider orders Lab tests

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Laboratory

4. Click “Quick Order Labs”

5. Select Patient Insurance, Location (South Dade Medical Centers), Physician, and Click Create

6. Click on Diagnosis Tab – Enter 2 Diagnosis for Lab Order (E03.9 – Hypothyroidism, E55.9 Vit. D

Deficiency)

7. Click on Referral/Orders/Lab Tab – Select Lab Tab below

8. Select “Routine” - Click Build

9. Select TSH and Vitamin D Level

10. Click OK – Link to DX’s

11. Click Save and Close

Correct



Comments:


Comments:

Rating:






Task 5: Change Laboratory Order (45 seconds)


A Lab order may need update or additional instructions after created.

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Laboratory

4. Click “Instructions” tab

5. Select most recent lab order

6. Click Care Event ID and Edit lab order instructions

7. Click Build

8. Click Urinalysis, Click OK

9. Click OK, Save and Close

Correct



Comments:


Comments:

Rating:






Task 6: Access Laboratory Order (20 seconds)


A Lab order may need update or additional instructions after created.

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Laboratory

4. Click “Instructions” tab

5. Review active lab order instructions history

Correct



Comments:


Comments:

Rating:






Task 7: Create Imaging (Radiology) Order (60 seconds)


After examining a Patient, it is decided to order a chest x-ray

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Orders

4. Click “Quick Order X-Ray”

5. Select Patient Insurance, Location (South Dade Medical Centers), Physician, and click Create

6. Click on Diagnosis Tab and Select Diagnosis for Referral being ordered (R07.9 Chest Pain)

7. Click on Referral\Orders\Lab Tab – Select Referral Tab shown below

8. Select “Routine”

9. Select Radiology “Yes”

10. Click Build

11. Click Chest X-Ray

12. Click OK – Link to DX’s

13. Save and Close

Correct



Comments:


Comments:

Rating:






Task 8: Change Imaging (Radiology) Order (25 seconds)


After ordering an X-Ray, the urgency level may need change.

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Referrals

4. Click on Instructions

5. Click Care Event ID and Edit the referral order instructions

6. Click on the Edit Feature for the chest x-ray

7. Select “Urgent” Order Level

8. Click OK

9. Save and Close

Correct



Comments:


Comments:

Rating:






Task 9: Access Imaging (Radiology) Order (20 seconds)


Part of the daily routine, access and review Lab Imaging (Radiology) orders

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Referrals

4. Click Instructions

5. Open the most recent Care Event ID = Active Chest X-Ray Ordered

6. View details

7. Close

Correct



Comments:


Comments:

Rating:






Task 10: Adjust the severity level of Drug-Drug notifications (30 seconds)

Take the participant to the starting point for the task. Provide the user with credentials with sufficient rights to

change the Drug-Drug level interactions.

A user with the assigned security function may adjust the level of notifications for drug-drug interactions in the

office.

Success:

Easily completed


Not completed

Comments:


Optimal Path:

1. Logon on MDVita with the User and Password supplied by the administrator

2. Click File

3. Click Options

4. Click System Options

5. Select “Drug Interactions” tab

6. Set Display Level to “Moderate”

7. Check “Exclude Food Interactions”

8. Click Save

9. Click Close

Correct



Comments:


Comments:

Rating:






Task 11: Act upon drug-drug and drug-allergy interventions prior medication order completion (90

seconds)


While adding a medication for a patient respond to drug-drug and drug-allergy interventions

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Medications

4. Click Add Medication

5. Find Coumadin tab 1 mg and click Select

6. Review Drug alert interventions

7. Check Acknowledge

8. Click Save


10. Find Bactrim medication and click Select

11. Review drug Allergic Reaction listed in interventions

12. Click Cancel

13. Review drug was not ordered

Correct



Comments:


Comments:

Rating:






Task 12: Record Patient’s Demographics (90 seconds)


Need to enter a new Patient:

Joan Arc, DOB: 12/24/1959, Female, Preferred Language: English, Sexual Orientation: Straight, Gender

Identity: Female, Race: White, Ethnicity: Not Hispanic or Latino

Success:

Easily completed


Not completed

Comments:


Optimal Path:

1. Add Medical Records

2. Enter Patient Name ‘Joan Arc’

3. Enter Address 7205 Corporate Drive, Miami FL 33126

4. Enter date of birth 12/24/1959

5. Enter Gender Female

6. Select Preferred Language English

7. Select Sexual Orientation Straight

8. Select Gender Identity Female

9. Click Tab ‘Additional Info’

10. Select Race1 White

11. Select Ethnicity1 Not Hispanic or Latino

12. Click Save

Correct



Comments:


Comments:

Rating:






Task 13: Change Patient’s Demographics (45 seconds)


Need to change patient demographics

Success:

Easily completed


Not completed

Comments:


Optimal Path:


2. Search for Patient ‘Joan Arc’ and click Select

3. Click Profile

4. Change date of birth to 12/24/1958

5. Change Preferred Language to Spanish

6. Click Additional Info tab

7. Change Rac1 to Decline to Specify

8. Click Save

Correct



Comments:


Comments:

Rating:






Task 14: Access Patient’s Demographics (30 seconds)


Need to access Patient demographics

Success:

Easily completed


Not completed

Comments:


Optimal Path:


2. Search for Patient ‘Joan Arc’ and click Select

3. Click Profile

4. Review demographics

Correct



Comments:


Comments:

Rating:






Task 15: Record problem in Problem List (30 seconds)


Need to record patient’s list of current problems

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Problem List

4. Click Add Problem

5. Find G47.00 - Insomnia and click Select

6. Enter 1/1/2017 Start date, Status Active

7. Click Save

Correct



Comments:


Comments:

Rating:






Task 16: Change Problem List (30 seconds)


Need to modify recorded patient’s list of current problems

Success:

Easily completed


Not completed

Comments:


Optimal Path:




4. Find Insomnia diagnosis

5. Change status from Active to Resolved

6. Enter modified date on 11/20/2017

7. Click Save

Correct



Comments:


Comments:

Rating:






Task 17: Access Problem List (20 seconds)


Patient arrives for a visit and wants to see a list of Active Problems

Success:

Easily completed


Not completed

Comments:


Optimal Path:




4. Click Active Only

5. Review problem list

Correct



Comments:


Comments:

Rating:






Task 18: Record medication in Medication List (30 seconds)


Need to record patient’s list of current medications

Success:

Easily completed


Not completed

Comments:


Optimal Path:





5. Find Protonix 20 MG and click Select

6. Enter 1/1/2017 Start date

7. Check Acknowledge Interventions

8. Click Save

Correct



Comments:


Comments:

Rating:






Task 19: Change Medication List (20 seconds)


Need to modify recorded patient’s list of current medications

Success:

Easily completed


Not completed

Comments:


Optimal Path:




4. Find Protonix medication and expand history

5. Click Edit in the latest medication instance

6. Update SIG details to “Once daily”

7. Enter Days: 30, Qty:30

8. Check Acknowledge Interventions notifications

9. Click Save

Correct



Comments:


Comments:

Rating:






Task 20: Access Medication List (15 seconds)


Patient arrives for a visit and wants to see what medications he\she is currently taking.

Success:

Easily completed


Not completed

Comments:


Optimal Path:




4. Click Active Only

5. Review medication list

Correct



Comments:


Comments:

Rating:






Task 21: Record medication allergy (30 seconds)


Seeing a patient and need to record patient’s allergies

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Allergies

4. Click Add

5. Select Tab ‘Drugs’

6. Select Penicillin in the list

7. Verify the Status is set to ‘Active’

8. Enter Today’s date in Allergy Date.

9. Select reactions: Respiratory Distress and Swelling

10. Click Save

Correct



Comments:


Comments:

Rating:






Task 22: Change Medication Allergy (20 seconds)


Seeing a patient and need to change the allergy details

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Allergies

4. Find Penicillin allergy and click Edit

5. Remove ‘Respiratory Distress’ reaction and add ‘Watery eyes’

6. Click Save

Correct



Comments:


Comments:

Rating:






Task 23: Access Patient’s Allergy List (15 seconds)


The patient arrives for a visit. Verify with the patient that the allergy list is current.

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Allergies

4. Review patient allergies

Correct



Comments:


Comments:

Rating:






Task 24: Configure CDS interventions (30 seconds)

Take the participant to the starting point for the task. The user account requires specific security in order to

perform this task.

Need to setup users for which CDS interventions are enabled

Success:

Easily completed


Not completed

Comments:


Optimal Path:

1. From main menu click ‘File’

2. Click ‘Security’

3. Click ‘Users’

4. Select your user account and Edit

5. Select ‘CDS’ tab

6. Enable the following Clinical CDS rules: Malignant Hypertension (Problem), Prescribing Selegiline

(Medication), Penicillin (Medication Allergy), Elevated bilirubin (Lab Test) , Fall prevention over 90

years of age (Demographics), Hypertension in very elderly (Vitals), and Potentially Inappropriate

Medication use in Older Adults (Combination)

7. Enable ‘Transition of Care’ and ‘Show References’ for rules above

8. Enable ‘Access to Diagnostic and Therapeutic resources’

9. Click Save and Close the screen

Correct



Comments:


Comments:

Rating:






Task 25: Act upon Problem List interventions (30 seconds)


Review and acting upon CDS alerts while entering a specific problem (Diagnosis)

Success:

Easily completed


Not completed

Comments:


Optimal Path:


2. Search for Patient ‘John Smith’ and click Select

3. Click ‘Problem List’

4. Click Add

5. Search for ICD10: I10

6. Click ‘Select’

7. Save

8. Open CDS alert, review intervention/resource information , and click ‘Handled’

9. Click on ‘D&T Resources’

10. Check Problem entered and click ‘Search’

11. Review Diagnosis and Therapeutic materials

12. Click ‘Close’

Correct



Comments:


Comments:

Rating:






Task 26: Act upon Medication List interventions (30 seconds)


Review and acting upon CDS alerts while entering a specific medication

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click ‘Medications’

4. Click Add

5. Search for ‘Selegiline Tab 5 MG’


7. Check ‘Acknowledge and Continue’ for alerts

8. Save


10. Click Close

Correct



Comments:


Comments:

Rating:






Task 27: Act upon Medication Allergy List interventions (35 seconds)


Review and acting upon CDS alerts while entering a specific medication to which the patient is allergic to.

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click ‘Allergies’

4. Click Add

5. Select Cephalosporins

6. Enter Allergy Date 1/1/2017

7. Select Reaction ‘Agitation’

8. Click ‘Save’


10. Click ‘Close’

Correct



Comments:


Comments:

Rating:






Task 28: Record Patient’s Implantable Device (60 seconds)


Record information about a Patient’s implantable device

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click ‘Implants’

4. Click ‘Add’

5. Enter an implantable device UDI, supplied by proctor

(01)00643169007222(17)160128(21)BLC200461H

6. Click Find

7. Enter today’s date in Start Date

8. Click Save

Correct



Comments:


Comments:

Rating:






Task 29: Change Patient’s Implantable Device (20 seconds)


Need to change patient demographics

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Implants

4. Change Status to ‘Inactive’

Correct



Comments:


Comments:

Rating:






Task 30: Access Patient’s Implantable Device (20 seconds)


Need to access Patient demographics

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Implants

4. Review Implantable Device list

Correct



Comments:


Comments:

Rating:






Task 31: Reconcile patient medication, problem, and medication allergy lists with a CCDA (120 seconds)


Import patient medication, problem, and medication allergy lists from a CCDA file received from another

provider

Success:

Easily completed


Not completed Comments:


Optimal Path:


2. Search for Patient ‘John Matters’ and click Select

3. Click ‘Import – Patient Information’ from the contextual menu

4. Click Transfer of Care

5. Got to directory ‘D:\MDVita\UsabilityTest’ and select file: MATTERS, JOHN.CDA.xml

6. Review document received

7. Click ‘Save’

8. Answer ‘Yes’ to the question ‘Do you want to incorporate document data?”

9. Allergy List is shown. Reconcile Allergy List selecting ‘Add Record’ from drop down

10. Click Incorporate

11. Review resulting Allergy list

12. Click ‘Confirm and Save’

13. Select Medication List

14. Reconcile Medication selecting ‘Add Record’

15. Click ‘Incorporate’

16. Review the resulting Medication list.


18. Select Problem List

19. Reconcile Problem List selecting ‘Add Record’


21. Review the resulting Problem list


23. Click Close

24. Open Patient ‘John Matters’ chart

25. Review Medication, Problem and Allergy lists

26. Click Export Medical Record

27. Save resulting CCDA

Correct



Comments:


Comments:

Rating:





Task 32: Act upon Demographics intervention (45 seconds)


Review and acting upon CDS alerts while entering patient’s demographics

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Profile

4. Change Date of Birth to 01/01/1924

5. Click Save

6. Open CDS alert, review intervention/resource information ,

7. Review bibliographic citation, developer, funding source, and release/revision date

8. click ‘Handled’

9. Click Close

Correct



Comments:


Comments:

Rating:






Task 33: Act upon Laboratory test intervention (50 seconds)


Review and acting upon CDS alerts while entering a specific lab test result

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click Laboratory

4. Click Add

5. Enter a new laboratory result using LOINC 1975-2 (Bilirubin) and value 1.93

6. Click Save


8. Click Close

Correct



Comments:


Comments:

Rating:






Task 34: Act upon Vital Signs intervention (35 seconds)


Review and acting upon CDS alerts while entering patient vital signs

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click care Events

4. Click Add

5. Enter Blood Pressure values 170/99

6. Click Save


8. Click Close

Correct



Comments:


Comments:

Rating:






Task 35: Act upon combination (Demographics and Medication) intervention (30 seconds)


Review and acting upon CDS alerts while entering patient Medication and triggering a CDS combined rule of

Medication and Demographics (Age).

Success:

Easily completed


Not completed

Comments:


Optimal Path:



3. Click ‘Medications’

4. Click Add

5. Search for ‘Benadryl 25 MG’


7. Check ‘Acknowledge and Continue’ for alerts

8. Click Save


10. Click Close

Correct



Comments:


Comments:

Rating:






Task 36: Act upon CCDA reconciliation interventions (130 seconds)


Import patient medication, problem, and medication allergy lists from a CCDA file received from another

provider and act upon triggered interventions during reconciliation

Success:

Easily completed


Not completed Comments:


Optimal Path:


2. Search for Patient ‘Aurora Smith’ and click Select

3. Click ‘Import – Patient Information’ from the contextual menu

4. Click Transfer of Care

5. Got to directory ‘D:\MDVita\UsabilityTest’ and select file: CDS.CDA.xml

6. Review document received

7. Click ‘Save’

8. Answer ‘Yes’ to the question ‘Do you want to incorporate document data?”

9. Allergy List is shown. Reconcile Allergy List selecting ‘Add Record’ from drop down

10. Click Incorporate

11. Review resulting Allergy list



14. Select Medication List

15. Reconcile Medication selecting ‘Add Record’


17. Review the resulting Medication list.



20. Select Problem List

21. Reconcile Problem List selecting ‘Add Record’


23. Review the resulting Problem list



26. Click Close

Correct



Comments:


Comments:

Rating:




Final Questions (15 Minutes)

What was your overall impression of this system?

What aspects of the system did you like most?

What aspects of the system did you like least?

Were there any features that you were surprised to see?

What features did you expect to encounter but did not see? That is, is there anything that is missing in this

application?

Compare this system to other systems you have used.

Would you recommend this system to your colleagues?



Appendix 6: SYSTEM USABILITY SCALE QUESTIONNAIRE In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems usability”

known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have elaborated on the SUS over

the years. Computation of the SUS score can be found in Brooke’s paper, in at

http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).

Strongly Strongly

disagree agree

1. I think that I would like to use this system frequently

2. I found the system unnecessarily complex

3. I thought the system was easy to use

4. I think that I would need the support of

a technical person to be able to use this

system

5. I found the various functions in this system were well integrated

6. I thought there was too much

inconsistency in this system

7. I would imagine that most people would learn to use this system very quickly

8. I found the system very cumbersome to use

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

http://www.usabilitynet.org/trump/documents/Suschapt.doc

http://www.usabilitynet.org/trump/documents/Suschapt.doc



9. I felt very confident using the system

9. I needed to learn a lot of things before I could get going with this system

1 2 3 4 5

1 2 3 4 5

MDVita - Drummond Group LLC...Electronic Health Record Usability Testing (EHRUT) Report of MDVita...

Documents

Transcript of MDVita - Drummond Group LLC...Electronic Health Record Usability Testing (EHRUT) Report of MDVita...