Page | 1

EHR Usability Test Report of WestCare Clinical Data System Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports

WestCare Clinical Data System Version 1

Date of Usability Test: 12/10/2019 – 12/13/2019 Date of Report: 12/13/2019 Report Prepared By: WestCare Foundation Inc. Diane Ludens Director of Software Development, Employee 702-385-2090 diane.ludens@westcare.com 1711 Whitney Mesa Dr. Henderson, NV 89014

Page | 2

EXECUTIVE SUMMARY

A usability test of WestCare Clinical Data System Version 1, Behavioral Health EHR was conducted between 12/10/2019 – 12/13/2019 over BlueJeans video conferencing by WestCare. The purpose of this test was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (WestCare Clinical Data System). During the usability test, 10 healthcare providers and other intended users matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 3 tasks typically conducted on an EHR:

Record a patient’s preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity

Change the patient’s preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity

Display the patient’s changed preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender ident ity

During the 30 minute one-on-one usability test, each participant was greeted by the administrator and asked to submit his/her signed non-disclosure agreement and informed consent/release form; they were instructed that they could withdraw at any time. Participants had prior experience with the EHR. The administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHRUT. During the testing, the administrator timed the test and recorded user performance data on paper and electronically. Participant video conferencing sessions were recorded for subsequent analysis. The following types of data were collected for each participant:

Number of tasks successfully completed within the allotted time without assistance

Time to complete the tasks

Number and types of errors

Path deviations

Participant’s verbalizations

Participant’s satisfaction ratings of the system All participant data was de-identified – no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire. Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT.

Task N Task Success

Path Deviation Task Time Errors Task Ratings 5=Easy

Measure

# Mean (SD)

Deviations (Observed / Optimal)

Mean (SD) in seconds

Deviations (Observed / Optimal)

Mean (SD)

Mean (SD)

Record a patient’s preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity

10

10 No deviations 177.3 No deviations 0.1 4.6

Page | 3

Change the patient’s preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity

10

10 No deviations 99.9 No deviations 0 4.8

Display the patient’s changed preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity

10

10 No deviations 19.4 No deviations 0 4.7

The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be: 94% In addition to the performance data, the following qualitative observations were made: - Major findings

Users demonstrated a good understanding of the mechanics of the navigation

Users had challenges locating the fields on the page due to the amount of information collected on the form

The time to complete the 2nd test was greatly reduced from the 1st test after users were more familiar with the screen

Users liked the ease of use and the drop-down functionality - Areas for improvement

Additional and ongoing training would help with the familiarity of the system

A more organized, less cluttered design would help with the data entry time

INTRODUCTION

The EHRUT tested for this study was WestCare Clinical Data System Version 1, Behavioral Health EHR. Designed to present medical information to healthcare providers in behavioral health facilities, the EHRUT consists of clinical data related to the client progression through treatment as well as billing data for reporting and collections from funders. The usability testing attempted to represent realistic exercises and conditions.

The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR Under Test (EHRUT). To this end, measures of effectiveness, efficiency and user satisfaction, such as difficulty rating and time on task, were captured during the usability testing.

METHOD PARTICIPANTS

A total of 10 participants were tested on the EHRUT. Participants in the test were clinicians, office managers, support personnel and administrative directors. Participants were recruited by Diane Ludens, WestCare. Participants were given the opportunity to have the same orientation and level of training as the actual end users would have received. For the test purposes, end-user characteristics were recruited and identified by the tester through email contact with the area directors at some of the larger WestCare facilities. Recruited participants had a mix of backgrounds and demographic characteristics. The following is a table of participants by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities

Page | 4

Participant Identifier

Participant Gender

Participant Age

Participant Education

Participant Occupation/Role

Participant Professional Experience

Participant Computer Experience

Participant Product Experience

Participant Assistive Technology Needs

01 Female 50-59 Bachelor's Degree

Quality Assurance Analyst 240 240 5 No

02 Female 20-29 Master's Degree

Director of Accreditation 84 180 72 No

03 Female 20-29

Some college credit, no degree

Application Support 24 24 12 No

04 Female 50-59 Bachelor's Degree Office Manager 360 264 48 No

05 Female 30-39

Some college credit, no degree Director of Billing 216 300 108 No

06 Female 30-39 Master's Degree Clinical Coordinator 60 240 18 No

07 Female 30-39 Bachelor's Degree Office Manager 120 180 84 No

08 Male 30-39 Bachelor's Degree

Admissions Coordinator 240 300 24 No

09 Male 30-39 Bachelor's Degree

Project Manager/IT Trainer 84 180 18 No

10 Female 50-59 Master's Degree

Project Manager/IT Trainer 336 264 45 No

10 participants (matching the demographics in the section on Participants) were recruited and 10 participated in the usability test. 0

Page | 5

participants failed to show for the study. Participants were scheduled for 20 minute sessions with 10 minutes in between each session for debrief by the administrator/data logger, and to load the video conferencing session. A spreadsheet was used to keep track of the participant schedule, and included each participant’s demographic characteristics as provided by the pre-test questionnaire.

STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well – that is, effectively, efficiently, and with satisfaction – and areas where the application failed to meet the needs of the participants. The data from this test may serve as a baseline for future tests with an updated version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made. During the usability test, participants interacted with 1 EHR. Each participant used the system in their location with the same EHR website portal and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant:

Number of tasks successfully completed within the allotted time without assistance

Time to complete the tasks

Number and types of errors

Path deviations

Participant’s verbalizations (comments)

Participant’s satisfaction ratings of the system

TASKS

A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including:

1. Record a patient’s preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity. 2. Change the patient’s preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity. 3. Display the patient’s changed preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity.

PROCEDURES

Upon arrival to the video conference session, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were then assigned a participant ID. Each participant reviewed and signed an informed consent and release form. The usability testing staff conducting the test was an experienced usability practitioner with 15 years of experience with the WestCare EHR, a degree in Computer Information System and the Director of Software Development.

The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments. The administrator also served as the data logger and took notes on task success, path deviations, number and type of errors, and comments, with the help of the playback of the video/audio recording. Participants were instructed to perform the tasks as efficiently as possible making as few errors and deviations as possible. For each task, the participants were given a written copy of the task. Task timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had successfully completed the task. Scoring is discussed below in Section 3.9. Following the session, the administrator gave the participant the post-test questionnaire and thanked each individual for their participation. Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet. Participants were thanked for their time.

Page | 6

TEST LOCATION

The test facility was the participant’s quiet office with a table, computer for the participant, and the BlueJeans video conferencing software for recording. Only the participant and administrator were in the test session. To ensure that the environment was comfortable for users, noise levels were kept to a minimum.

TEST ENVIRONMENT

The EHRUT would be typically be used in a healthcare office or facility.

In this instance, the testing was conducted in a WestCare office facility. For testing, the computer used a PC or laptop running Microsoft Windows 10.

The participants used a mouse or touchpad and keyboard when interacting with the EHRUT. The WestCare Clinical Data System used a variety of screen sizes with standard resolutions and color settings appropriate for their PC/laptop. The application was set up by the participant by opening their browser. The application itself was running on a Linux platform using a PostGreSQL database on a WAN connection. Technically, the system performance (i.e., response time) was representative to what actual users would experience in a field implementation.

TEST FORMS AND TOOLS

During the usability test, various documents and instruments were used, including: 1. Non-Disclosure Agreement and Informed Consent [Appendix B1] 2. Preliminary Questions [Appendix B2] 3. Moderator’s Guide [Appendix B3] 4. Post-test Questionnaire [Appendix B4] 5. System Usability Scale Questionnaire [Appendix B5] The participant’s interaction with the EHRUT and his/her verbal comments were captured and recorded digitally with audio/video conferencing software running over BlueJeans. The test session were electronically transmitted to the BlueJeans portal/cloud.

PARTICIPANT INSTRUCTIONS

The administrator reads the following instructions aloud to the each participant: See the full moderator’s guide in Appendix B3

USABILITY METRICS

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: 1. Effectiveness of WestCare Clinical Data System by measuring participant success rates and errors 2. Efficiency of WestCare Clinical Data System by measuring the average task time and path deviations 3. Satisfaction with WestCare Clinical Data System by measuring ease of use ratings

Page | 7

DATA SCORING The following table details how tasks were scored, errors evaluated, and the time data analyzed.

Rationale and Scoring

Effectiveness: Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis. The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage. Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency. Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when constructing tasks. Target task times used for task times in the Moderator’s Guide must be operationally defined by taking multiple measures of optimal performance and multiplying by some factor [e.g., 1.25] that allows some time buffer because the participants are presumably not trained to expert performance. Thus, if expert, optimal performance on a task was [x] seconds then allotted task time performance was [x * 1.25] seconds. This ratio should be aggregated across tasks and reported with mean and variance scores.

Effectiveness: Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a “Failure.” No task times were taken for errors. The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors. This should also be expressed as the mean number of failed tasks per participant. On a qualitative level, an enumeration of errors and error types should be collected.

Efficiency: Task Deviations

The participant’s path (i.e., steps) through the application was recorded. Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation.

Efficiency: Task Time

Each task was timed from when the administrator said “Begin” until the participant said, “Done.” If he or she failed to say “Done,” the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.

Satisfaction: Task Rating

Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. 12

Common convention is that average ratings for systems judged easy to use should be 3.3 or above. To measure participants’ confidence in and likeability of the WestCare Clinical Data System overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently,” “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See full System Usability Score questionnaire in Appendix B5.

Page | 8

RESULTS DATA ANALYSIS AND REPORTING

The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses.

The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based on performance with these tasks to be: 94%.

The users found the system to be intuitive and easy to use for the most part.

RISK RATING

In the event that a user makes an error in the workflow for entering demographics, this is the determination that was made regarding level of risk to patient safety for each task. On a scale of 1-5 with 1 being riskiest and 5 being least risky: Record a patient’s preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity : 3 Change the patient’s preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity : 4 Display the patient’s changed preferred language, date of birth, birth sex, race, ethnicity, sexual orientation, gender identity: 5 The tasks decline in risk due to processes in place to review and, if necessary, update demographics as the client progresses through the system of care.

EFFECTIVENESS Based on the success, failure and path deviation data, the system was always effective in allowing the demographics to be completed.

EFFICIENCY Based on the observations of the task time and deviation data, the first task was the most labor intensive, third step the least and second step markedly improved from the first task due to the learned familiarity with the page layout.

SATISFACTION Based on the task ratings and SUS results data, users are largely highly satisfied with the system.

MAJOR FINDINGS

The risk to patient safety in the addition/modification and viewing of demographic information is determined to be low. The area that may pose a risk is that the information is not entered methodically and with precision. The biggest challenges for users were: locating each individual data element on the page, locating the proper response from the drop-downs, misspelling words and mistyping numbers. This can be mitigated by providing training to users. The more familiar that the user gets with the screen, the lower the risk becomes.

AREAS FOR IMPROVEMENT

The biggest area for improvement would be to arrange the data on the page so that it is less cluttered and easier to find what you’re looking for. End-user training could also help close this gap.

Page | 9

APPENDICES

Appendix A: PARTICIPANT DEMOGRAPHICS

Following is a high-level overview of the participants in this study. Gender Men 2 Women 8 Total (participants) 10

Occupation/Role Quality Assurance Analyst 1 Director of Accreditation 1 Application Support 1 Office Manager 2 Director of Billing Clinical Coordinator Admission Coordinator Project Manager/IT Trainer

1 1 1 2

Total (participants) 10

Years of Experience Years experience (average) 3.6 Facility Use of EHR – Demographics a5 All paper 0 Some paper, some electronic 0%

All electronic 100%

Page | 10

Appendix B1: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM

Page | 11

Appendix B2: PRELIMINARY QUESTIONS

Page | 12

Appendix B3: MODERATOR’S GUIDE

Page | 13

EHRUT Usability Test Moderator’s Guide Thank you for participating in this study. Your input is very important. Our session today will last about 30 minutes. During that time you will use an instance of an electronic health record. I will ask you to complete a few tasks using this system and answer some questions. Please try to complete the tasks on your own following the instructions very closely. Please note that we are not testing you we are testing the system, therefore if you have difficulty all this means is that something needs to be improved in the system. Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Please be honest with your opinions. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Should you feel it necessary you are able to withdraw at any time during the testing. Do you have any questions or concerns? Following the procedural instructions, participants opened up screen share and the EHR and as their first task and the administrator gave the following instructions: For each task, I will describe the task to you and ask you to begin. I will ask you your impressions about the task once you are done. Participants were then given 3 tasks to complete. Administrator/Data Logger Date Time Participant # Prior to testing

Confirm schedule with Participants

Ensure EHRUT is running properly

Ensure BlueJeans recording equipment is running properly Prior to each participant:

Start session recordings with BlueJeans After each participant:

End session recordings with BlueJeans After all testing

Back up all video and data files

Page | 14

Page | 15

Appendix B4: POST-TEST QUESTIONNAIRE

Page | 16

Appendix B5: SYSTEM USABILITY SCALE QUESTIONNAIRE