Usability Test Report for HealthShare 2013 · 2019-09-14 · Participants were asked to complete...

of 27

Usability Test Report for HealthShare 2013.1

Table of Contents Executive Summary ....................................................................................................................................... 2

Introduction .................................................................................................................................................. 4

Method ......................................................................................................................................................... 4

Participants ............................................................................................................................................... 4

Study Design.............................................................................................................................................. 6

Tasks .......................................................................................................................................................... 7

Procedures ................................................................................................................................................ 7

Test Environment ...................................................................................................................................... 7

Participant Instructions ............................................................................................................................. 7

Data Scoring .............................................................................................................................................. 8

Results ........................................................................................................................................................... 9

Task by Task Results .................................................................................................................................. 9

Major Findings ........................................................................................................................................ 18

Areas for Improvement ........................................................................................................................... 19

Appendix A – Participant Documentation and Feedback Form .................................................................. 20

Appendix B – Complete Numeric Results of HealthShare Usability Test .................................................... 26

of 27

Executive Summary A usability test of HealthShare 2013.1, a strategic informatics platform often used to create and view an

aggregated patient record, was conducted on September 10-12, 2013. The test was executed from

various sites by logging into a centrally configuration solution. This ensured participation by a collection

of representative users from various backgrounds and regions.

The study assessed the usability of searching for and identifying a patient, viewing the medication list

and viewing the allergy list. For each step in the process, the time was recorded at each step.

Participants were given an outline, introducing them to the purpose of the test, and were instructed to

complete a series of tests and document their results in a form, found in Appendix A. Participants were

not given assistance in navigating the test.

When beginning the test, the system presented the user with a unique ID that was recorded on their

form. This allowed the administrator to cross reference the recorded observations with the timing

recorded in the test session.

The following data was requested or recorded:

Time to execute each task

Time to complete the entire series of tests

Path deviations

Participant impressions and notes on usability

Participant satisfaction ratings

The following background data was collected on each participant:

Clinical Experience, role and experience level

Prior Experience with this system

Location

Technical details of the solution used for access – internet browser and operating system

The participant’s identity was not associated with the collected data. The system recorded data was

matched to the data recorded by the participant via a unique session ID created during the testing

session. No compensation was given to participants.

A total of twenty (20) participants completed the test.

Based on the established criteria for Effectiveness, Efficiency and Satisfaction, all tasks were considered

successful .

of 27

Efficiency Satisfaction Effectiveness

Task Time Goal

Avg Time (s)

Errors Avg Time Reported (s)

Avg Deviations

Average

Login N/A Not applicable 9.05

Patient Search 20 12.59 0 8.95 Patient Select 30 11.08 0.1

5 8.65

View Summary 60 30.52 0 8.4 View Allergies 60 34.05 0.0

5 8.85

View Meds 90 57.47 0.65

8.35 Logout N/A Not applicable 9.35

Figure 1: Summarized Results of Usability Test

0 12.59 0 11.08 0 30.52 0 34.05 0 57.47

of 27

Introduction The Electronic Health Record Under Test (EHRUT) in this usability study was HealthShare 2013.1.

Designed to aggregate and present composite health records of a single patient to a clinician,

HealthShare is used in varied clinical settings. Typically used to create an aggregated record on a

regional, state-wide or national level, the system is not used as the day-to-day EHR system by a clinician

but is used to view aggregated records for patients who may be receiving care from multiple settings.

Accordingly, HealthShare does not allow the manual entry of data by clinicians. Recording and updating

of clinical data, including medication allergies and medications, is done electronically by submitting data

to HealthShare in a standard or custom format, such as HL7v2, CCD, CDA, etc. Therefore, usability

testing centered around accessing the data in an effective, efficient and satisfactory manner.

The purpose of the test was to verify that a typical clinical user can access the patient record and access

the specific clinical data for medication allergies and medications. Both subjective and objective

measures were captured in the areas of effectiveness, efficiency and satisfaction.

Method

Participants A total of 20 participants were tested on the EHRUT. Participants in this test were Physicians,

Pharmacists, Nurses and Laboratory or Radiology Technologists, by background. All participants are

employees of InterSystems working in varied offices around the world and on different products.

Recruited participants had a mix of clinical backgrounds and reside in different regions around the

world. Participants were given one day to login and complete the test.

Participants resided in the United States (6), Australia (9), Germany (1), United Arab Emirates (2), United

Kingdom (1) and Brazil (1).

Figure 2: Participant Breakdown by Region

of 27

Participants were asked to use their own systems to connect to the server and complete the test.

Eleven (11) of the 20 participants used Windows, while nine (9) use Apple (MacOS) based systems. For

Windows users, Internet Explorer was the predominant browser used by eight (8) of the participants.

The other Windows-based participants used Firefox (2) and Chrome (1). For the MacOS based

participants, Safari was the predominant browser choice, representing five (5) of the responses with the

others using Chrome (2) and Firefox (2).

Figure 3: Client Operating System and Browser Details

The clinical background and years of experience was varied. Participants all had a clinical background as

most users of the system for viewing clinical data would have a clinical background. Participants were

either a physician (8), nurse (7), pharmacist (3) or laboratory/radiology technician (2). All users currently

work in IT, but some still practice as well.

of 27

Figure 4: Participant Breakdown by Clinical Background

Years of experience in clinical practice ranged from 0 to 30 years, with the collective average being 15.45

years.

Study Design The goal of this test was to ensure the typical clinical user can intuitively access the system, identify the

patient and access the clinical data for medication allergies and medication allergy list. The study is

designed to uncover any areas of usability challenges which have a negative impact on the effectiveness,

efficiency and user satisfaction of the system.

The data collected here may serve as a baseline for future testing.

Participants executed the tasks from their own environments remotely and without supervision. All

participants were fluent in English and were provided with the same instructions and feedback form,

written in English. The system logged times for execution of each task. Feedback forms were submitted

following the test.

The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected

and analyzed for each participant:

Tasks completed successfully

Time to complete tasks

Number and types of errors

Path deviations

Participant’s comments

Participant’s ratings on effectiveness, efficiency and satisfaction with the system

of 27

Tasks

The following tasks were outlined to be representative of the kinds of activities a clinical user would

perform in the EHRUT:

1. Login to the EHRUT

2. Search for a specific patient

3. Select that patient to view the patient record

4. Review the summary chart for the patient

5. View the patient’s allergy list

6. View the patient’s medication list

7. Logout to end the session

Procedures Participants were given instructions and time frames within which to complete the test via email. An

automatically generated ID uniquely identifying the test session is created and provided on login.

Following test completion, participants completed the feedback form and submitted it to the

administrators of the test.

Participants were instructed to perform the outlined tasks as quickly as possible, while taking time to

process and understand the clinical data, as they would in a real care setting.

The Task instructions were provided and participants were advised to print these instructions for

efficient execution.

Participants were advised to complete the feedback form following the completion of all seven tasks.

Participant details, task times, deviations and comments were recorded into a spreadsheet.

Test Environment The EHRUT would typically be accessed from a care setting, such as a hospital or clinic, using a

supported web browser. For this test, the environment was accessed remotely via supported web

browser.

The EHRUT solution was running on Windows, hosted in Amazon Cloud Services, allowing for remote

access by the participants.

The variance of browsers, access locations, and system operating systems is representative of a typical

regional health information exchange which would use the EHRUT. The system performance should also

be representative of a typical setup.

Participant Instructions Participants received a simple email asking them to participate within a day. The instructions for

accessing and completing the test were included in a Word doc supplied electronically to participants.

This contained an explanation of the test, access instructions, the test script and feedback form. This

HealthShare Usability Testing document is included in Appendix A.

of 27

Data Scoring

To rate the system for effectiveness and efficiency, time measures were taken as each participant

navigated the tasks.

Effectiveness: Task success was deemed to be the successful completion of the task, without error and

without a reasonable amount of time for that task. Task failure was deemed to be if the participant

abandoned the task, did not succeed at the task within a reasonable amount of time or received and

error.

Efficiency: Task success was deemed to be if the participant was able to complete the task quickly and

efficiently without deviations. Task failure was deemed to be if the participant was not able to complete

the task without deviations quickly. Average and max/min times were calculated.

Satisfaction: For each task, the participant was asked to use a likert scale from 1-10 to indicate their

satisfaction with the solution for that task, with 1 being Not Satisfied and 10 being Very Satisfied. A

score of 7 was considered acceptable.

of 27

Results

Task by Task Results Participants were asked to complete seven steps in this usability test.

1. Login to the EHRUT






7. Logout to end the session

For each step, the system collected data as the participant navigated the system, including time for

completion of each step, as well as any diversions. For each test item, the participant responded to a

feedback form, following the test which allowed them to provide a satisfaction rating (on a likert scale

from 1-10), as well as add addition comments on any difficulties they experienced while executing the

test and any comments on the usability of this portion of the software.

Steps 1 and 7 (Login and Logout) were not scored for effectiveness and efficiency since they were not

related to the clinical use of the system being evaluated. Login and Logout were only scored on

Satisfaction.

The complete spreadsheet of results is attached in Appendix B – Usability Testing Results. In this matrix,

all participants are identified by an anonymous session ID which was created for their session. Users

were presented with this session ID when they accessed the system for the first time and recorded it on

their feedback form. The matrix does not contain a sequential set of IDs as administrative users were

testing/using the system in the interim and generated IDs as well. Those IDs were eliminated and only

those tests from clinical participants who submitted a feedback form were used in the collected data.

Results for Task 1: Login

Only Satisfaction was tracked for the Login task. On a scale of 1-10, Login received an average score of

9.05 with standard deviation of 1.54. The minimum score was 5, while the maximum score was 10.

Average 9.05

Standard Deviation 1.54

Min 5.00

Max 10.00

With a goal of 7 for satisfaction, Login was deemed to be successful.

of 27

Results for Task2: Patient Search

In this test, participants were asked to search for a specific patient – John Smith. No training on the use

of the user interface was given. This test was scored on efficiency, effectiveness and satisfaction.

The task was considered to be efficient if it was successfully completed in under twenty (20) seconds

with less than an average of 0.5 deviations across the participant set.

The task was considered to be effective if it was completed without error and in under twenty (20)

seconds. To evaluate this, a combination of automatically calculated data, along with user response

data regarding errors was evaluated.

Below are the complete data, across the twenty participants for the Patient Search test.

Patient Search

Time (Search)

Deviations (Search)

Satisfaction (Search)

11.607 0 10

11.326 0 10

9.641 0 10

15.99 0 8

18.798 0 10

0 7

13.167 0 8

9.89 0 10

11.232 0 10

14.633 0 7

4.056 0 9

11.435 0 10

8.658 0 8

10.436 0 7

17.254 0 10

15.288 0 8

10.312 0 10

9.095 0 10

16.973 0 10

19.329 0 7

Time Deviations Satisfaction

Goals 20.00 0.50 7.00

Average 12.59 0.00 8.95

Standard Deviation

3.95

0.00

1.28

Min 4.06 0.00 7.00

Max 19.33 0.00 10.00

of 27

As noted above, the average time to complete the task was 12.59 seconds, well under the goal of 20

seconds. The max across the participants was 19.33 seconds. No deviations were recorded across the

entire participant set.

The Satisfaction scoring showed an average of 8.95, well above the 7.0 goal. Standard deviation on

satisfaction scoring was 1.28 with minimum recorded score of 7 and maximum recorded score at 10.

No errors were reported in completing this task by the twenty participants. A few participants indicated

a slow response time, but this was considered to be related to internet speed and not software

response time as slow response times were reported across the test set for certain participants, but the

system performed very well for others in different locations.

Accordingly, this task was deemed to meet acceptable standards for efficiency, effectiveness and

satisfaction.

Results for Task 3: Patient Select

In this test, participants were asked to view the results of their search and select the specified patient,

John Smith. The test was scored on efficiency, effectiveness and satisfaction.

The task was considered to be efficient if it was successfully completed in under thirty (30) seconds with

less than an average of 0.5 deviations across the participant set. More time was allotted to this task

than the previous task is it is very important for the clinician to review search results and select the

correct patient.

The task was considered to be effective if it was completed without error and in under thirty (30)



Patient Select

Time (Select)

Deviations (Select)

Satisfaction (Select)

3.79 0 9

9.344 2 10

2.59 0 9

4.087 0 9

14.087 0 8

12.386 0 5

9.64 0 8

5.46 0 10

5.616 0 10

9.266 0 8

4.805 0 9

8.44 0 10

10.67 0 9

26.099 0 7

12.012 0 10

18.252 1 8

of 27

15.631 0 10

3.588 0 9

21.746 0 10

24.102 0 5


Goal 30.00 7.00

Average 11.08 0.15 8.65

Standard Deviation

7.03

0.49

1.53

Minimum 2.59 0.00 5.00

Maximum 26.10 2.00 10.00


seconds. The max across the participants was 26.10 seconds. Across all the participants, only two

participants deviated from the path, giving a deviation average for the test as 0.15.



No errors were reported in completing this task by the twenty participants. A few participants indicated

a slow response time, but this was considered to be related to internet speed and not software

response time as slow response times were reported across the test set for certain participants, but the

system performed very well for others in different locations.


satisfaction.

Results for Task 4: View Summary

In this test, participants were asked to view the summary of the clinical record for the patient. The test

was scored on efficiency, effectiveness and satisfaction.

The task was considered to be efficient if it was successfully completed in under sixty (60) seconds with

less than an average of one (1) deviations across the participant set. More time was allotted to this task

than the previous task as clinicians were not presented with clinical data previously and may review it

for a longer period of time.

The task was considered to be effective if it was completed without error and in under sixty (60)



View Summary

Time (Summary)

Deviations (Summary)

Satisfaction (Summary)

47.861 0 7

11.419 0 10

of 27

11.716 0 10

41.855 0 9

19.204 0 10

44.335 0 5

6.849 0 10

38.61 0 10

26.598 0 10

30.155 0 8

3.245 0 9

12.636 0 10

12.293 0 8

0 6

21.107 0 6

37.3 0 8

17.285 0 10

17.846 0 7

22.683 0 10

156.843 0 5


Goals 60.00 7.00

Average 30.52 0.00 8.40

Standard Deviation

33.33

0.00

1.82

Minimum 3.25 0.00 5.00

Maximum 156.84 0.00 10.00


seconds. The max across the participants was 156.84 seconds. The standard deviation was 33.33,

minimum time was 3.25 and maximum time was 156.84. On analyzing across all tests, it was noticed

that the final participant (identified in the results matrix in Appendix B as Participant Session 44) took a

much larger than average amount of time on every test. As there was no active monitoring of the test,

it is unknown if this participant was distracted or multi-tasking. For reference and comparison to the

goal, the time ratings when eliminating Participant 44 and using the data from the other 19 participants

results in the following time data:

Average 23.50

Std Dev 13.61

Min 3.25

Max 47.86

No participants deviated from the test. No errors were reported when executing this task.

of 27

Based on the overall average (with all 20 participants data being used in the calculations) of 30.52, as

well as the lack of deviations, this task is considered to be efficient. The time data, along with the lack of

errors reported, indicates this task is effective.




satisfaction.

Results for Task 5: View Allergy List

In this test, participants were asked to view the allergy list for the patient by clicking on the Allergies and

Alerts tab in the clinical record. The test was scored on efficiency, effectiveness and satisfaction.

The task was considered to be efficient if it was successfully completed in under sixty (60) seconds with

less than an average of one (1) deviations across the participant set. More time was allotted to this task

than the previous task as clinicians are reviewing clinical data.

The task was considered to be effective if it was completed without error and in under sixty (60)



View Allergies

Time (Allergies)

Deviations (Allergies)

Satisfaction (Allergies)

44.491 0 8

21.637 0 10

12.433 0 10

80.293 0 9

3.089 0 10

61.76 0 6

14.976 0 10

12.605 0 10

36.535 0 10

52.088 0 6

24.367 0 8

8.175 0 10

30.17 0 10

1 7

11.746 0 10

39.031 0 7

35.583 0 10

5.101 0 10

9.329 0 10

143.567 0 6


Goals 60.00 7.00

of 27

Average 34.05 0.05 8.85

Standard Deviation

33.74

0.22 1.60

Min 3.09 0.00

6.00

Max 143.57 1.00 10.00


seconds. The max across the participants was 143.57 seconds. The standard deviation was 33.74,

minimum time was 3.09 and maximum time was 143.57. On analyzing across all tests, it was noticed

that the final participant (identified in the results matrix in Appendix B as Participant Session 44) took a

much larger than average amount of time on every test. As there was no active monitoring of the test,

it is unknown if this participant was distracted or multi-tasking. For reference and comparison to the

goal, the time ratings when eliminating Participant 44 and using the data from the other 19 participants

results in the following time data:

Average 27.97

Std Dev 21.47

Min 3.09

Max 80.29

This resulted in a reduced average of 29.97. One participant was still above our 60 second goal.

However, with all other participants performing the task in significantly less than the allotted time, the

task is considered to have met its time goal.

Only one participant deviated from the test and made only a single deviation. No errors were reported

when executing this task.


well as the adjusted average of 27.97 seconds when removing data from Participant 44, the task met its

time goal for efficiency. Combined with the lack of deviations across the participant set (one deviation

in total for 0.05 average), this task is considered to be efficient. The time data, along with the lack of

errors reported, indicates this task is effective.




satisfaction.

Results for Task 6: View Medication List

In this test, participants were asked to view the medication list for the patient by clicking on the

Medications tab in the clinical record. The test was scored on efficiency, effectiveness and satisfaction.

of 27

The task was considered to be efficient if it was successfully completed in under ninety (90) seconds

with less than an average of one (1) deviations across the participant set. More time was allotted for

this task as reviewing Medication History is a lengthier process than allergies. Recent and Historical

Medications are displayed and there is more data to view.

The task was considered to be effective if it was completed without error and in under ninety (90)



View Meds

Time (Meds)

Deviations (Meds)

Satisfaction (Meds)

47.861 0 3

48.267 2 8

17.347 0 9

25.085 0 9

13.93 0 10

118.373 0 6

9.796 0 10

23.993 0 10

32.308 0 10

99.294 0 7

30.014 0 8

56.877 0 10

59.171 0 10

1 6

87.158 8 10

42.447 0 8

3.525 1 10

151.196 0 7

21.247 1 10

203.985 0 6


Goal 90.00 7.00

Average 57.47 0.6

5 8.35

Standard Deviation

53.00

1.81

1.98

Min 3.53 0.00

3.00

Max 203.99 8.00 10.00

As noted above, the average time to complete the task was 57.47 seconds, which is under the goal of 90

seconds. The max across the participants was 203.99 seconds, while the minimum was 3.53. The

standard deviation was 53.00. On analyzing across all tests, it was noticed that the final participant

(identified in the results matrix in Appendix B as Participant Session 44) took a much larger than average

amount of time on every test. As there was no active monitoring of the test, it is unknown if this

of 27

participant was distracted or multi-tasking. For reference and comparison to the goal, the time ratings

when eliminating Participant 44 and using the data from the other 19 participants results in the

following time data:

Average 49.33

Std Dev 40.51

Min 3.53

Max 151.20

This resulted in a reduced average of 49.33. Four participants, in total, took longer than 90 seconds to

complete the task, while the other 16 participants completed the task in under the 90 second goal Based

on the average, the task is considered to have met its time goal.

Five out of twenty (5/20) participants deviated from the task, resulting in 0.65 deviations across the

task. No errors were reported when executing this task.


well as the adjusted average of 49.33 seconds when removing data from Participant 44, the task met its

time goal for efficiency. Combined with the low number of average deviations across the participant

set, this task is considered to be efficient. The time data, along with the lack of errors reported,

indicates this task is effective.




satisfaction.

Commentary on the Medication section gave feedback on how medication presentation could be

organized more effectively for different care settings. This is described further in the Areas for

Improvement section.

Results for Task 7: Logout

Only Satisfaction was tracked for the Logout task. On a scale of 1-10, Login received an average score of

9.05 with standard deviation of 1.54. The minimum score was 5, while the maximum score was 10.

Average 9.35

Standard Deviation 0.93

Min 7.00

Max 10.00

With a goal of 7 for satisfaction, Logout was deemed to be successful with an average score of 9.35 and

a minimum score of 7.0.

of 27

Major Findings

Based on the collection of feedback, the tasks were all successful and indicated the system is quite

usable for searching and selecting a patient record, viewing medication allergies and viewing a patients

medication list. While minor suggestions were logged (noted in the Areas for Improvement section), the

average satisfaction scores indicated the system is currently usable in a clinical setting.

of 27

Areas for Improvement

Written feedback was collected on each task item. Upon reviewing this data, areas of improvement

were identified in some of the tasks. In each of these areas, further analysis with the clinical team to

focus on specific areas will be undertaken.

Patient Search

A few participants commented that there were a large number of potential search fields and suggested

that fewer search options be provided for the average user or that the form be sectioned to draw the

user’s attention to the most commonly used areas for search – e.g. Name or Date of Birth.

As this screen is completely customizable by site, reducing and reorganizing the fields is part of the

system setup and configuration. However, it will be placed under consideration to generally reduce the

default number of fields shown.

View Allergy List

A few participants suggested that allergies for medication category could be highlighted differently than

other recorded allergies (e.g. food allergies). This will be considered for a future enhancement.

View Medication List

Many participants, depending on their care setting, suggested improvements to the Medication List,

including presentation of additional data on the main list (without navigating to details) and different

organization between recent and historical medications. Additional focus on the Medication List with a

clinical focus group will be planned.

of 27

Appendix A – Participant Documentation and Feedback Form

HealthShare Usability Testing

Thank you for agreeing to participate in our HealthShare Usability testing. Please read this complete

document, including script and form elements you will be responding to prior to beginning the test.

This test is focused specifically around accessing the medication list and allergy list for a specific patient.

During this test, we will be asking you to perform the following tasks:

8. Login to a HealthShare system






14. Logout of the session

While executing the tests, we would like you to evaluate the system on how satisfied are you with the

user interface. Were you satisfied with the execution of the task, how the information was presented

and how you would be able to use this data in a real-world setting?

The system will be recording metrics that time tasks and allow us to score the system on effectiveness

(could you complete the task) and efficiency (how much time did each task take).

The feedback form provided will ask you to give an evaluation for each of the tasks.

Additional feedback is welcome on the evaluation form as well.

of 27

Test and Task Outline

Now please execute the following tasks. The feedback form is available on the next screen.

VERY IMPORTANT: During the LOGIN task, you will receive a popup providing you with a session ID.

Please record this number on your feedback form.

Start: Access System Please access the HealthShare system at the following link:

http://54.243.145.157:57772/csp/healthshare/hsaccess/HS.UI.PatientSearch.cls

Task 1: Login to the system

Login ID: doctor

Password: doctor

Click Login

Following login, you will be given a session ID. Please record this ID on your feedback form!

Task 2: Search for the Patient

Search for a patient named John Smith

Click Search at the bottom of the search form.

Task 3: Select Patient

Select John Smith to view the patient’s record.

Task 4: View Summary Record for the Patient

You are presented with the summary of the most recent data for John Smith. Please view this screen

and ensure you can locate the allergies and medications.

Task 5: View Allergies for the Patient

Select the Allergies tab to display a chart of the known allergies for the patient.

Is the Allergies tab easy to access and understand?

Is it easy to identify which allergies are drug/medication related?

Task 6: View Medications for the Patient

Select the Medications tab to display a chart of the known allergies for the patient.

Is the Medication tab easy to access and understand?

Task 7: Logout

From the top menu bar, click Logout to end the session.

of 27

Feedback Form

You may fill this feedback form out manually and then return to [email protected] (An

electronically filled or a scanned copy is acceptable)

Test Date:

Participant Details

Clinical Role (e.g. Nurse, Doctor, Specialist, etc):

Years in Practice:

Do you have any prior experience with HealthShare?

Where are you located?

Which internet browser are you using (e.g. Firefox, IE, Chrome, Safari)?

What operating system are you using (e.g. Windows, Mac)?

Identification

Please record the Session ID that displayed on the screen following login:

Task Feedback

Please rate each of the tasks for effectiveness, efficiency and satisfaction, with 1 being least and 10

being greatest.

Task 1: Login

1. Were you able to login without difficulty?

2. Did you receive any errors while executing this task (if so, please specify)?

3. How satisfied were you with the system login process?

(Not satisfied) (Satisfied) (Very Satisfied)

1 2 3 4 5 6 7 8 9 10

4. Please provide any additional feedback:

mailto:[email protected]

of 27

Task 2: Search for Patient (John Smith)

1. Were you able to search for the patient without difficulty?


3. How satisfied were you with the patient search interface?


1 2 3 4 5 6 7 8 9 10


Task 3: Select Patient (John Smith)

1. Were you able to select for the patient without difficulty?


3. How satisfied were you with the patient selection interface?


1 2 3 4 5 6 7 8 9 10


Task 4: View Patient Summary

1. Were you able to view the patient summary without difficulty?


of 27

3. How satisfied were you with the patient summary interface?


1 2 3 4 5 6 7 8 9 10


Task 5: Access Allergy List

1. Were you able to access the Allergy List without difficulty?

2. Were you able to clearly identify which allergies pertained to medication/drug reactions?


4. How satisfied were you with the allergy list access?


1 2 3 4 5 6 7 8 9 10


Task 6: Access Medication List

1. Were you able to access the Medication List without difficulty?


3. How effective was the medication list access?

(Not effective) (Effective) (Very Effective)

1 2 3 4 5 6 7 8 9 10


of 27

Task 7: Logout

1. Were you able to access the Logout link without difficulty?

2. Were you successfully logged out and no longer had access to the patient record?


4. How satisfied were you with the logout process?


1 2 3 4 5 6 7 8 9 10


Appendix B – Complete Numeric Results of HealthShare Usability Test

Login Patient Search Patient Select View Summary View Allergies View Meds

Logout Anonymous

ID Clinical Role Yrs in

Practice Experience

with EHRUT

Location Browser OS Satisfaction (Login)

Time (Search)

Deviations (Search)

Satisfaction (Search)

Time (Select)

Deviations (Select)

Satisfaction (Select)

Time (Summary)

Deviations (Summary)

Satisfaction (Summary)

Time (Allergies)

Deviations (Allergies)

Satisfaction (Allergies)

Time (Meds)

Deviations (Meds)

Satisfaction (Meds)

Satisfaction (Logout)

Total Time

1

Physician

11

1

United States

Safari

MacOS

10

11.607

0

10

3.79

0

9

47.861

0

7

44.491

0

8

47.861

0

3

9

155.61

3 Nurse 22 0 Australia Firefox Windows 10 11.326 0 10 9.344 2 10 11.419 0 10 21.637 0 10 48.267 2 8 10 101.993

5

Pharmacist

10

3

United States

Safari

MacOS

10

9.641

0

10

2.59

0

9

11.716

0

10

12.433

0

10

17.347

0

9

10

53.727

15 Physician 23 2 Brazil Safari MacOS 8 15.99 0 8 4.087 0 9 41.855 0 9 80.293 0 9 25.085 0 9 10 167.31

17

Nurse

25

0

United Kingdom

IE

Windows

9

18.798

0

10

14.087

0

8

19.204

0

10

3.089

0

10

13.93

0

10

10

69.108

18 Lab Tech 15 0 Australia IE Windows 5 0 7 12.386 0 5 44.335 0 5 61.76 0 6 118.373 0 6 8 236.854

19

Midwife, Nurse

12

0

Australia

IE

Windows

10

13.167

0

8

9.64

0

8

6.849

0

10

14.976

0

10

9.796

0

10

8

54.428

20 Physician 30 2 Australia Chrome MacOS 10 9.89 0 10 5.46 0 10 38.61 0 10 12.605 0 10 23.993 0 10 10 90.558

21

Nurse

16

1

United States

Chrome

Windows

10

11.232

0

10

5.616

0

10

26.598

0

10

36.535

0

10

32.308

0

10

10

112.289

26 Physician 18 0 Australia Firefox MacOS 9 14.633 0 7 9.266 0 8 30.155 0 8 52.088 0 6 99.294 0 7 9 205.436

27

Physician

29

2

United States

Chrome

MacOS

9

4.056

0

9

4.805

0

9

3.245

0

9

24.367

0

8

30.014

0

8

9

66.487

28 Nurse 15 0 Germany Safari MacOS 10 11.435 0 10 8.44 0 10 12.636 0 10 8.175 0 10 56.877 0 10 10 97.563

30

Radiation Therapist

6

0

Australia

IE

Windows

10

8.658

0

8

10.67

0

9

12.293

0

8

30.17

0

10

59.171

0

10

10

120.962

31 Pharmacist 6 0 Australia IE Windows 5 10.436 0 7 26.099 0 7 0 6 1 7 1 6 8 36.535

34

Physician

0

1

United States

Safari

MacOS

10

17.254

0

10

12.012

0

10

21.107

0

6

11.746

0

10

87.158

8

10

10

149.277

35

Nurse

16

0

United States

IE

Windows

9

15.288

0

8

18.252

1

8

37.3

0

8

39.031

0

7

42.447

0

8

9

152.318

36 Nurse 29 1 Australia IE Windows 9 10.312 0 10 15.631 0 10 17.285 0 10 35.583 0 10 3.525 1 10 10 82.336

38

Physician

12

0

United Arab Emirites

Firefox

MacOS

10

9.095

0

10

3.588

0

9

17.846

0

7

5.101

0

10

151.196

0

7

10

186.826

40

Pharmacist

8

0

United Arab Emirites

IE

Windows

10

16.973

0

10

21.746

0

10

22.683

0

10

9.329

0

10

21.247

1

10

10

91.978

44 Physician 6 0 Australia Firefox Windows 8 19.329 0 7 24.102 0 5 156.843 0 5 143.567 0 6 203.985 0 6 7 547.826

Goal Average

Experience

15.45

0.65

Goal

7.00

20.00

0.50

7.00

30.00

7.00

60.00

7.00

60.00

7.00

90.00

7.00

7.00

Averages Average 9.05

12.59 0.00 8.95 11.08

0.15 8.65 30.52 0.00 8.40 34.05

0.05 8.85 57.47 0.65

8.35 9.35 138.97

Std Dev Total

Participants

20.00

Std Dev

1.54

3.95

0.00

1.28

7.03

0.49

1.53

33.33

0.00

1.82

33.74

0.22

1.60

53.00

1.81

1.98

0.93

110.44

Min Min 5.00

4.06 0.00 7.00 2.59 0.00 5.00 3.25 0.00 5.00 3.09 0.00

6.00 3.53 0.00

3.00 7.00 36.54

Max

Max

10.00

19.33

0.00

10.00

26.10

2.00

10.00

156.84

0.00

10.00

143.57

1.00

10.00

203.99

8.00

10.00

10.00

547.83

Page 26 of 27

Usability Test Report for HealthShare 2013 · 2019-09-14 · Participants were asked to complete...

Documents

Transcript of Usability Test Report for HealthShare 2013 · 2019-09-14 · Participants were asked to complete...