Project #5

Resource Management System (RMS)

Prototype & Pilot Testing

December 10, 2014 EDIT 732

Group E Candice Bowes Kimberlie Fair Vi Huynh Kara Pantalena Dina Saffouri Nathan Walby Nanchang Yang

EDIT 732: Fall 2014 2

Contents

Prototype and Pilot Testing Overview ......................................................................................... 3

Refined Project Concept Statement ..................................................................................... 3

Prototyping Process ............................................................................................................................ 3

Prototype Fidelity ........................................................................................................................ 4

Prototype Walkthrough ............................................................................................................. 4

Pilot Testing Process ........................................................................................................................ 10

Next Steps .................................................................................................................................. 10

Appendix A: RMS Pilot Testing Notes ....................................................................................... 11

EDIT 732: Fall 2014 3

Prototype and Pilot Testing Overview

We developed a click-through, medium fidelity, “T” prototype for our mobile application: Resource Management System (RMS). This initial RMS version 1.0 was designed based on the project team’s contextual analysis, system requirements and modeling, and wireframe design. RMS v1.0 was pilot tested to capture user experience, document user feedback, and identify any potential gaps in functionality or design. The project team will develop subsequent versions of the RMS mobile application by incorporating pilot testing data. The remaining sections of this document provide more detail on our prototype and pilot testing process.

Refined Project Concept Statement

RMS is a robust mobile application that supports the professional and academic activities of surgical and podiatry residents at INOVA hospitals. RMS serves as a repository of selected resources identified by both residents and residency program instructors. These resources are essential for the development of industry-leading general surgeons and podiatrists. RMS includes additional features that enable residents to manage their time and responsibilities, prepare for exams, and to give feedback to educators on how to improve the overall quality of education and training within the residency programs.

Prototyping Process The prototype process began with project team identifying a few key questions.

1. What core features and capabilities should the group include in this first prototype?

2. What level of fidelity is needed for RMS pilot testing? 3. To what level of complexity and functionality should specific features be

developed? 4. To develop the prototype, what type of tool is bested suited for design and user

testing? As the group discussed these questions, considerations were given to information from previous deliverables, available time to complete and test the prototype, and acknowledgement of future iterations. Once initial decisions were made, the prototype design process included the following set of actions:

1. Review of system requirements 2. Review of digital user storyboard for reinforcement of emotional impact

considerations 3. Review and use of wireframe designs to develop RMS capabilities and features 4. Review of industry best practices in mobile application design 5. Project team members independently designing RMS screens and features 6. Collaborative review and discussion of independent designs 7. Development of a singular RMS prototype

EDIT 732: Fall 2014 4

8. Project team review and minor changes

Prototype Fidelity To prepare for pilot testing, the project team progressively increased the fidelity of the prototype by adding appropriate complexity and depth to RMS v1.0 features. For example, initial low fidelity designs did not include enough functionality to simulate the user task of creating events. Once reviewed, the team added the appropriate sequence of user actions and screens to simulate this task. Ultimately, the team designed RMS v1.0 to reflect medium fidelity “T” prototype. The “T” prototype allows us to evaluate the general structure of the application and its usability, while allowing the user to drill down in specified tasks. In addition, RMS v1.0 prototype design included considerations on design industry best practices for selection of color, common actions on a mobile, and branding.

Prototype Walkthrough Figure A and B. Opening screens of RMS prototype. A represents the login screen and B shows the general structure.

When the user first encounters the application, they see a login screen (Figure A). This login screen is meant to allow for personalization of the resources. An example would be whether you are a general surgery resident or podiatry resident. These two different residencies have two different schedules and core resources. Personalization also allows for tracking of the resident by the administration and faculty. There are three main features of the application, as seen in Figure B. There is a calendar, resources, and feedback.

EDIT 732: Fall 2014 5

Figure C. The main calendar.

If the user selects the calendar, they see the view shown in Figure C. A month view of the calendar allows the user to see events in advance. Currently, residents do not know what they will be doing in lecture and lab until they arrive in ASTEC. Days with events are circled. If they select the circled date, they are taken to the daily planner, Figure D.

Figure D. Daily planner screen.

EDIT 732: Fall 2014 6

This daily planner will have auto-populated residency events, but the user can also add personal events (Figure E). Events linked to specific resources appear in blue. When selected, the user is taken to the linked files in the resources portion of the application (Figure F). Figure E. Add an event to the calendar. Figure F. Instructor resources page, linked to

required residency events.

Resources are organized within titled folders (ex. “Bone Fixations”) for ease of recognition and navigation. The icon next to the title of the file identifies the specific kind of file (an image, a folder, a video, or a PDF). The user can toggle between instructor resources and my resources (resources they have uploaded for personal reference). Note the search function in the top right. This function allows the user to search through resources within the application or on the web (Figure G).

EDIT 732: Fall 2014 7

Figure G. The search function.

EDIT 732: Fall 2014 8

Users are able to upload their own resources to this application. In the menu in the top left are extra options: add an event, add a resource, and recommend a resource. If they select to add a resource, they are able to upload content from their mobile device or from the web to access later (Figures H and I). When the user selects “Recommend a Resource,” they are able to send a resource to ASTEC for review and potential inclusion in the resource library.

Figure H and I. Add a resource selected from menu button.

EDIT 732: Fall 2014 9

The third part of the application is sending feedback to ASTEC (Figure J). Currently, ASTEC has paper surveys for residents to complete after each lecture and laboratory session. The residents often forget to turn them in and find it a burden to fill out a paper copy. ASTEC needs this feedback in order to apply for and maintain accreditation, track residents’ progress, and improve residency programs. Making it electronic makes it easier to collect the required data and easier for the residents to submit. Figure J. Feedback form.

EDIT 732: Fall 2014 10

Pilot Testing Process RMS v1.0 was pilot tested by both the project team and by a diverse group of primary and secondary users. From designers, clients, and end users, the project team deliberately selected this group of testers to gain a holistic perspective of the application. However, end user feedback will be prioritized and have greater influence on future RMS versions.

Appendix A: RMS Pilot Testing Notes contains the specific observations and notes gathered during the pilot testing. Overall the process of pilot testing included the following actions:

1. Publishing RMS v1.0 on a hosted server in HMTL output for highest fidelity testing using mobile devices.

2. Selecting pilot testers and scheduling user testing date. 3. Identification of specific tasks users will be asked to complete:

a. Simulate the process of finding resources linked to calendar events b. Simulate the process of adding resources c. Navigate to the Feedback Tool

4. Developing parameters to ensure minimal test administrator influence on user testing.

5. Developing a set of debrief questions for test administrator to ask. Examples of questions included:

a. What was easy to do? b. What was difficult? c. Why did you select certain buttons? Did those buttons take you to where

you expected? d. What were you thinking as you tried to complete various user tasks? e. Are there other features you would like to see? f. What recommendations do you have to improve this application?

6. Test Administrator facilitating a face-to-face observation of pilot testing. 7. Document and review of pilot testing notes for RMS v2.0 considerations.

Next Steps The project team plans to continue refinement of RMS v1.0 starting in January 2015.

EDIT 732: Fall 2014 11

Appendix A: RMS Pilot Testing Notes Date: 12/4/2014 Test Location: ASTEC Classroom Client/User Testers: Dr. Graling, Dr. Bachman, Dr. Dort, Franco Piscitani, Larry Walker, Dr. Esther Yim (Post-graduate year 3 resident) Secondary Testers: Kerianne Bowes, Robert Bowes

Testing Environment and Procedures The testing session began with Group E member sharing the link with testers. Testers opened the link on their own cellphones. Franco used the B-Line video system to record their explorations of the device. Audio recording was also captured. Team member observed testers in the use of the prototype and captured tester recommendations. User Feedback

1. The Enter button on the log in page would not work on Franco’s or Kerianne’s

Samsung phones, but did work on Larry’s Samsung phone. Reason unknown.

2. The prototype was generally well received. The group gave it a 3.5 on a 5 point

scale for ease of navigation.

3. Dr. Bachman noticed that the Search icon led to the Google page screenshot,

but then the back arrow led to the feedback/evaluation form. This was confusing

for her.

4. The color scheme was well received with a couple of exceptions. The gradient

gray was distracting to some and the back arrow on the gray background was not

clear to others. Dr. Bachman offered the RGB codes used in the INOVA

branding which are as follows.

a) Inova Blue #004B8D, R O G 44 B 119, PMS 288 C, C 100 M 67 Y O K 23

b) Inova Red #D52B1E, R 213 G 43 B 30, PMS 485 C, C O M 95 Y 100 K

O

c) Inova Light Blue #6CADDF, R 108 G 173 B 223, PMS 284 C, C 55 M 19 Y

O K O

5. The doctors pointed out that the resources embedded for the FLS lab were

procedural and not truly FLS skills.

a) They clarified that FLS skills are specific to the FLS curriculum and

involved such things a tying knots and other exercises related to manual

dexterity. Procedural skills are related to surgery.

6. The testers did find some things confusing in the design. They are listed below.

a) The Back button was slow and many did not recognize it as being a “back

button” upon first sight.

b) They didn’t like the clicking sounds. (Artifact of Captivate)

c) They thought the calendar icon was a calculator at first, but liked the

calendar features of the daily planner and the option for personalization.

d) They were confused by the “hamburger” menu vs. the back button in

terms of functionality.

EDIT 732: Fall 2014 12

e) One tester said that the app was “not intuitive” and would have liked an

intro screen prior to first use.

f) All testers indicated a dislike for the rotation on the FLS screen which

would only stay in place if they locked the rotation button on their phones

and seemed to feel that doing so would be cumbersome in real life.

g) One tester commented that the menu button does different things in

different locations and suggested that the options to add an event or

resource be on a separate screen.

Suggestions for Improvements Dr. Dort

1. Dr. Dort suggested including the APS (SCORE) curriculum as a resource.

2. Including such books as Netter’s Anatomy as resources, although this may

involve copyright and compatibility (Kindle vs. iBooks) issues. Suggestion to

consider adding an e-reader to the app was made to possibly resolve such

issues.

Dr. Yim

3. The resident tester, Dr. Yim, she would like a “Share” feature to allow sharing of

resources between residents.

4. Dr. Yim also wanted to know if an Internet connection would be required to use

the features not directly dependent on such a connection. She said the Wi-Fi at

the hospital was “iffy”.

5. Dr. Yim also wanted to know if there could be the option of leaving the app open

once the user logs in to avoid having to log in repeatedly or with each use.

6. Dr. Yim thought the feedback button was a chat feature and was surprised when

it took her to the evaluation form. She wondered if a chat feature that would

NOT be accessible by anyone other than the sender and the receiver would be

possible. She said that if it was viewable by others, they would likely not use it.

7. Dr. Yim also expressed the desire for an app search feature to facilitate the

quick location of specific resources.

Dr. Bachman

8. Dr. Bachman suggested changing “Instructor Resources” to “Assigned

Resources” to avoid confusion regarding for whom the resources found therein

were for. She thought at first that they were for the instructors because of the

name.

9. Dr. Bachman also indicated that residents text each other ALL THE TIME, so a

chat feature may be worth including.

10. Dr. Bachman also thought that a FAQ page would be helpful.

11. Dr. Bachman wanted to know if it would be possible to create cohorts of

residents to facilitate sending the appropriate resources and evaluation forms to

EDIT 732: Fall 2014 13

the appropriate groups. She wanted to know how to do it from the back end of

the app and that instruction on that would be needed.

12. Dr. Bachman also expressed a desire to be able to keep PDFs and other

resources on her computer at her desk as opposed to storing them on her

phone. This was projected to be the desire of the resident users as well.

13. Dr. Bachman also indicated that “Add a Resource” needs to be linked to an

event and that an App Search feature separate from the Internet Search feature

would be helpful.

14. Dr. Bachman also indicated that a queue with all the individual residents’

pending evaluation forms be included.

15. Dr. Bachman also expressed the desire for a Notes feature and a picture

capture feature both of which could then be added to “My Resources”.