AsthmaMD Usability Test Report

AsthmaMD Usability Test Report

Ann Junker Sup Suh

February 18, 2018

1

Table of Contents

Executive Summary 2

Research Questions 2

Methodology 2

Task List 4

Test Environment 5

Researcher Roles 5

Evaluation Measures 6

Participant Characteristics 7

Results 8

Findings & Recommendations 15

Appendix 21 Collected Data 21 Adult Consent to Participate in Research 30 Screener questions 31 Moderator Guide 32 Observation Guide 38 Post Testing Questions 41

2

Executive Summary AsthmaMD is a mobile application to help Asthma patients to manage, record and share symptoms with doctors and caregivers. In previous heuristic and cognitive evaluations, we found several common issues; such as poor labeling and lack of system feedback. Due to these, navigation was difficult and user expectations were not met. In usability study tested three core tasks: 1) Set up reminder 2) Enter Data and 3) Share record with Doctor. We conducted tests using iOS app on iPhone x. The mobile screen interaction and testing sessions were video recorded. We observed both quantitative and qualitative data such as errors, completion rate, ease of use, satisfaction ratings, facial expressions, and quotes.

The study was helpful in answering our research questions and also helped us find out the sources of the errors. We found common issues in navigation and confirmation. Placement and design of buttons confused the users with navigation and observed confidence level when participants. We recommend implementing consistent placement of save, back and cancel buttons across the app to improve navigation. We also recommend giving users more feedback when entering and saving changes. In addition, two out of four participants had hard time discovering a menu item in global navigation. We recommend differentiating color of the global navigation with background color to improve discoverability.

Research Questions This study further tested issues found in previous evaluations. Can users easily and successfully set up medication reminders?

● Do users understand how to change the time of a reminder? ● Did users successfully change medication frequency? ● Did users understand that the change was saved?

Can users easily and successfully log an asthma attack?

● Was navigation for entering data intuitive? ● Do users understand that information was entered for the log? ● Do users notice the button to save entered information?

Can users easily and successfully share their asthma record with their doctor?

● Do users understand how to navigation to Send Report screen? ● Do users interact with selection buttons for contacts? ● Do users understand what data is being sent?

Methodology

3

This was an assessment test for a usability study. We collected data on the number of errors, task success rate, common user behaviors and comments, and qualitative data on the user’s experience, impressions, and attitudes. Test Design We planned to use within-subjects design, testing with two user groups (based on severity of asthma). Though we had participants with severe asthma, their schedule changed last minute. To include the four participants we planned to test, we needed to quickly find replacement participants, one did not have asthma. Each participant completed the task using the think outloud protocol. Participants completed the tasks in the same order.

1. Change medication reminder 2. Log asthma attack 3. Share asthma record with physician

The user account, medications, physician's name, and two entries for Enter Data were pre-populated on the Asthma MD application prior to usability testing. This allowed for consistency between each users' tasks, and allowed users to focus on the tasks being studied rather than the set up for those tasks. Procedure The test sessions took approximately 30 minutes.

● Introduction (5 minutes) ○ When participant arrives, they were greeted and shown to the study facilities ○ Reviewed and asked participant to sign consent form ○ Reviewed purpose of the study, researcher’s roles, participant’s role, and study

format with participant ● Warm-up Questions (5 minutes)

○ Put participant at ease ○ Confirm information gathered from the invitation screening survey ○ Gathered additional demographic information ○ Learned about participants current activities in asthma management

● Task-based Assessment (15 minutes) ○ Think aloud protocol to understand participant's thoughts and gathered quotes ○ Moderator read tasks outloud and handed out task sheets to participant ○ Participants asked to go through three scenarios/tasks ○ Post-task questions to gather feedback

● Wrap-up Questionnaire and Questions (5 minutes) ○ Gathered scoring information of participant's experience of tested tasks; such as

satisfaction and ease of use ○ Gathered comments to clarify issues and understand need of product

● Moderator, Participant Guides, and Observer

4

○ Ensured consistency between tests ○ Ensured all tests have necessary forms and information available

Task List This study tested three tasks common to regular user interactions with the application as related to their asthma symptoms. The following areas of the application were studied, along with their scenarios.

Task 1 Change medication reminder

Scenario You just had a doctor visit and the doctor suggests you to take Alvesco at 10pm everyday, once a day, instead of 10am. Use AsthmaMD app to change the medication reminder settings.

State AsthmaMD was started on the Settings page, requiring the user to navigate to the reminder page.

Machine iPhone X with AsthmaMD application

Successful completion criteria Changing reminder time of Alvesco to once a day at 10am

Benchmark Filled in 100% of content and mistakes < 1

Task 2 Log an asthma attack

Scenario Imagine you just had an asthma attack after you went for a long run. You took both rescue and control medications but your symptom is worse than usual with frequent coughs and peak flow at 400 l/min. Now you want to record information related to this asthma attack to help you keep track of asthma symptoms over time. Open AsthmaMD to log this asthma attack.

State AsthmaMD was started on last participant was last on, requiring the user to navigate to the Enter Data page.


Successful completion criteria Successfully sends the record to Dr. Grey

Benchmark Filled in 100% of content and mistakes < 1

5

Task 3 Share asthma record with physician

Scenario Imagine you have a doctor visit coming up in two days to talk about changes in the frequency of your asthma symptoms. You want to send data you have logged in the past on the app to discuss with Dr. Grey. Use AsthmaMD app to send your report to Dr. Grey.

State AsthmaMD was started on last participant was last on, requiring the user to navigate to the Diary-chart page.


Successful completion criteria Entering specified data from Scenario. Clicking on Save Record.

Benchmark Mistakes < 1

Test Environment Each session consisted of two researchers (one moderator and one notetaker) and a test participant. We planned to conduct the tests with all researchers and test participants meeting at a quiet, convenient location of their choosing. Though weather constraints changed plans. The moderator was with the participants, and the observer watched remotely through Google Hangouts. Each participant used an iPhone X to interact with the app. The iPhone was connected to a laptop using Reflector 2 to allow the observer to see and to record the actions of the participant to review later. The researchers recorded audio conversation and video with the mirrored iPhone screen. At one-point Hangouts disconnected during one task for one participant, although it was recovered quickly.

Researcher Roles Moderator The moderator facilitated the test. After the initial greeting and introduction, the moderator was the only researcher communicating with the participant. Using the moderator guide, the moderator set the scenario and gave the participant tasks to complete, reminding participant to think outloud if necessary. After the participant fully attempted to complete a task, the moderator probed about any verbal or behavioral feedback observed during the task. If the participant struggled to complete a task or became frustrated, the moderator guided or

6

encouraged the participant once the participant’s initial feedback was noted. The moderator set up AsthmaMD as specified in the setup checklist in between test sessions. The moderator observed both the Reflector 2 phone screen as well as the participant’s interaction with the application, verbal feedback, behavior, and gestures. Notetaker The notetaker observed and took notes of participant's interaction with the application, verbal feedback, behavior, and gestures. Notetaker did not interact with the participant. Moderator and Notetaker debriefed after all test sessions were complete to review their notes. They looked for similarities in observations and participant comments to clarify thoughts, organize notes, and create priorities in issues found.

Evaluation Measures In order to answer the questions outlined in “Research Questions” above, we collected both Quantitative and Qualitative data during the test. Quantitative

Task Completion Rate

Success Participants completed task without mistake and filled in all the content

Partial Success Participants did not input all the content but completed task or made 1-3 mistakes

Failure Participants could not complete the tasks or had 4 or more mistakes

Mistakes

● Number of clicks on incorrect menu item ● Does not change “once a day” label on Reminder page ● Does not uncheck other contact list on Send Report page ● Does not click Save to complete tasks ● Does not click send report to complete task ● Does not change time for reminder ● Number of data omissions to enter data for Enter Data page ● Number of wrong data entry on Enter Data page

7

Ease-of-use ratings ● Post task questions with likert scale (1 - very difficult to 5 - very easy)

Satisfaction ratings

● Post test questions with likert scale (1 - very difficult to 5 - very easy) Qualitative Attitude and Emotion during the task observations During our first observation we found our original method of logging attitude and emotion needed adjustment. We initially had three categories; confident and neutral, Unsure and confused, and not confident and frustrated. The following table is our updated categorization.

Quotes to Observe (think out loud)

Facial Expression, Emotions

Confident "This was easy" "I was able to do that" “I am done with the task”

smiling, confident, nodding

Neutral N/A Neutral

Unsure, Confused “I think I did it right” “Am I done?” “I was confused” “I was not sure”

Frowning, tilt head to the side

Not Confident, Frustrated

"I don't know what to do" “I was frustrated” "I can't figure this out"

Frustration, shaking their head, sighing, unsure

Post-task questions with participant comments

● Each post task questionnaire asked participants why they chose the ease-of-use score. We observed their comments and identified specific sources of the problems.

Post-study wrap-up questions

● We asked general usability of the app by asking what they liked and disliked about the app and identify specific problems and recommendations.

Participant Characteristics We planned to observed four participants with mild to moderate asthma and those with severe asthma and how they interacted with and relate to the application. As mentioned earlier Though as schedules changes, we needed to adjust plans as follows:

8

Characteristic Number of Participants

Age 21-30 31-40 41-50 51-60

2 1 1 0

Use Smartphone Yes

All

Severity of Asthma No Asthma Mild to Moderate asthmatic symptoms Severe asthmatic symptoms

1 4 0

Doctor visits due to asthma symptoms Annual check up N/A

3 0

Currently asthma management Have Inhaler nearby N/A

3 0

Education Level Bachelors Graduate school

2 2

Participants were recruited through friends and were sent an email reminder of their scheduled time. Since participants could not self-select their asthma severity, they were asked to complete a screener questionnaire to help place them in a characteristics group.

Characteristics group placement

We used the Mayo Clinic Classification system to place participants in a group. ● Mild to Moderate - Symptoms two to seven days days a week ● Severe - "Symptoms throughout the day on most days"

Results In previous heuristic and cognitive evaluations, we found several common issues; such as poor labeling and lack of system feedback. Due to these, navigation was difficult and user expectations were not met. These results provide quantitative and qualitative data to understand the user perspective better. Raw data is provided in the appendix.

https://www.mayoclinic.org/diseases-conditions/asthma/diagnosis-treatment/drc-20369660

9

Task 1: Set up medication reminder Research Questions Can users easily and successfully set up medication reminders?

● Do users understand how to change the time of a reminder? ● Did users successfully change medication frequency? ● Did users understand that the change was saved?

Completion rate The overall success rate was mixed. Based on observation and number of mistakes recorded:

● 2 Successes ● 1 Partial Success ● 1 Failure

General results Throughout the task - 3 of 16 quotes and facial expressions were confident. 6 quotes and facial expressions were of frustration, indicating they were unable to find what they were looking for and wished there was more confirmation of actions. One additional issue found (outside research questions) during testing was related to navigating to start the task.

● 2 of 4 participants had errors of navigating to other screens before going to the Reminders screen. Both participants indicated they did not see the taskbar at the bottom of the screen.

● 3 of 4 participants rated the predictability of where in the app to start this task as 4 or 5 (easy or very easy).

● 1 of 4 participants' quotes and facial expressions indicated confusion. 1 of 4 participants attitude and facial expressions indicated frustration. 2 participants were neutral.

Understanding how to change the time of a reminder ● 4 of 4 participants had no errors with changing the time of the reminder. ● 4 of 4 participants rated the ease recognizing the app

recorded the information entered as expected as 4 or 5 (easy or very easy).

● 3 of 4 participants' quotes and facial expression were neutral. 1 participant was frustrated, indicating "I don't see a time to take it."

Changing medication frequency

● 2 of 4 participants had errors changing the frequency. ○ 1 participant did not complete this step

10

○ 1 participant took 4 tries to complete this task - they looked under the time setting for frequency instead of seeing the buttons to change frequency

● 3 or 4 participants rated the efficient use of icons and labels guided them complete the task as 4 or 5 (easy or very easy).

● 1 of 4 participants' quotes indicated frustration in not finding how to do this step. 1 participants' quote indicated he wasn't sure about switching frequency. The other 2 were neutral.

Understanding the change was saved

● 4 of 4 participants had no errors in saving their changes. ● 3 or 4 participants rated the efficient use of icons and labels guided them complete the

task as 4 or 5 (easy or very easy). One participant wanting a save button. ● 1 of 4 participants' quotes and facial express indicated confident in saving information. 3

of 4 were neutral (one saying they wished there was more confirmation).

Task 2: Log an asthma attack Research Questions Can users easily and successfully log an asthma attack?

● Was navigation for entering data intuitive? ● Do users understand that information was entered for the log? ● Do users notice the button to save entered information?

Completion Rate The overall success rate was mostly partial success. Based on observation and number of mistakes recorded:

11

● 1 Success ● 3 Partial Successes

General results

● 12 of 19 participants' quotes and facial expressions were unsure or not confident throughout the task.

Navigating to entering data was intuitive

● Finding where to start the task ○ 4 of 4 participants had no errors in navigating to the Enter Data Screen. ○ 4 of 4 participants rated the predictability of where in the app to start this task

as 4 or 5 (easy or very easy). Participants indicated the button in the task bar was clear.

○ 2 of 4 participants quotes' and facial expressions were unsure or not confident when starting the task. One was looking around app to become familiar with it before deciding which screen to use.

● Navigating during entering data ○ 2 of 4 participants had a total of 2 errors each in

navigating between screens. ○ 4 of 4 participants rated the ease recognizing

the app recorded the information entered as expected as 4 or 5 (easy or very easy). One participant indicated the Enter Data button was confusing and didn't know right away if it was the back/save button.

○ 4 of 4 participants' quotes and facial expressions were neutral to frustrated through the different steps on entering data. Participants indicated they were guessing if the action was the right one before taking it.

Understanding that information was entered for the log (see screen shot above for Navigating during entering data)

● 1 of 4 participants had a commission for entering a symptom.

● 1 of 4 participants did not enter a Trigger. ● 4 or 4 participants rated the efficient use of icons and

labels guided them complete the task as 4 or 5 (easy or very easy). One participant who rated it a 4 indicated it was because you need to know what things mean and get familiar with the app before using.

● 2 of 4 participants' quotes indicated they were not sure how to log this as an attack versus taking normal medication.

12

Noticing the button to save entered information ● 4 of 4 participants had no errors. ● No specific ratings were given here, though other notes

indicate the save button was not visible enough. ● 3 of 4 participants' quotes and facial expressions were neutral

or confident. 1 of 4 users was confused about the dialog box that opened after saving.

Task 3: Share asthma record with physician Research Questions Can users easily and successfully share their asthma record with their doctor?

● Do users understand how to navigation to Send Report screen? ● Do users interact with selection buttons for contacts? ● Do user understand what data is being sent?

Completion rate The overall success rate was mostly successes, though since this is the last task we believe that was some learning effect happening. Based on observation and number of mistakes recorded:

● 3 Successes ● 1 Partial Success

General results

13

● 4 of 4 participants completed the task quicker than expected, as perceived by observers. ● 4 or 4 participants rated the efficient use of icons and labels guided them complete the

task as 4 or 5 (easy or very easy). Participants indicated the buttons were clear. ● 7 of 9 participants' quotes and facial expressions were confident, overall. None indicated

frustration. Navigating to Send Report screen

● Participants did not have errors in navigating to the Send Report screen ● 4 of 4 participants rated the predictability of where in the app to start this task as 5

(very easy). Participants mentioned that based on knowledge from previous tasks, they assumed where to navigate to.

● 1 of 4 participants' quotes and facial expressions of being unsure of where to navigate to. All others were neutral or confident.

Interacting with data (selecting how to send report to)

● 1 of 4 participants did not uncheck people they were not supposed to send the report to and missed reading the instructions above.

● 4 of 4 participants rated the ease recognizing the app recorded the information entered as expected as 5 (very easy). Participants indicated they saw the dialog feedback and knew the report was sent.

● 4 of 4 participants' quotes and facial expressions of being neutral or confident.

Understanding effect of action One additional question asked on the survey was to find out if the participants knew what information was being sent in the report.

● 2 of 4 participants mentioned they were unsure of how the data would be received.

● 3 of 4 participants mentioned they thought the information sent was based on the date range selected in the screen previous to the send report screen.

● 2 of 4 participants' quotes and facial expressions of being unsure about what was sent. The other 2 participants felt sure about was being sent.

14

Post-test Results Overall participants felt the app was something they would use or recommend to users. Though 2 of 4 participants with mild asthma didn't feel they would use because they don't feel the need to track their symptoms, "I wouldn't use frequently because I don't need too much." Though, they felt it would be useful for others, "would suggest to others" and "I liked you can send before a doctor visit."

1 - Strongly disagree

Disagree Neither disagree or

agree

Agree 5- Strongly agree

I felt AsthmaMD would fit my asthma symptom tracking needs

P2 P4 P1 P3

I would use AsthmaMD frequently

P2 P3 P1 P4

I would like to use AsthmaMD with my doctor

P2 P1 P3 P4

I would recommend AsthmaMD to others

P2 P3

P1 P4

Some final participant quotes about the tasks: ● P1 - I take care of a lot of seniors from families and I think it would be more difficult for

them.

15

● P2 - When I was younger I took rescue 2-3 day, and that would be more useful. But not useful for me (has mild asthma now).

● P4 - I like data and being able to run reports and send to others. ● P3 - Labels at the bottom were not intuitive, but I was able to understand with context. ● P1 - We didn't do every screen but I think symptoms page adding a class for type of attack

(severity) a check box would be helpful. ● P2 - Having different profiles for a mom who is managing their children's asthma.

Findings & Recommendations Positive Findings Overall, we found the participants did not have issues some specific tasks, issues found in the heuristic evaluation and cognitive walkthrough. Participant 4 mentioned “I get a confirmation” after sending report to doctor, and we observed confidence in participants when feedback was given. We recommend implementing this feedback across other tasks such as saving data and changing medication reminders. We anticipated that participants might have hard time finding where to send report, as the menu is not visible under global task bar. However, all participants successfully found the path to send report and we recommend keeping this information architecture. Usability Issues and Recommendations We found several usability issues and organized the recommendations in the order of priority. Priority of items was chosen based on the ease of development, impact of reducing users errors and frustration, and how many areas of the application were affected, with 5 being the highest priority and 1 being the lowest.

16

Implement review before sending sensitive information

Source or Error The task required participants to send the medical report to Dr. Grey, but one of the participants did not uncheck other contacts and send the record to all his contacts. Priority This has priority of 5, as it could be a problem or embarrassment if the users accidentally send private health information to unwanted contacts. It also has a relatively easy fix. Recommendation We recommend implementing a review pop up confirming user’s action to prevent errors.

17

Improve Navigation for Entering Data

Source or Error Users had problem with navigation. The navigation is designed so that it makes users go back and forth between the main screen and each item in the main page. The navigation creates multiple steps. Users also had placement and the design of the Enter Data button for navigation. P3 initially had hard time finding this button. P1 said "I wasn't sure the enter data was a back button and if it would save," and P4 said "I guess I click on enter data [button at the top left of screen]." Priority This has priority of 4, as we observed most errors and difficulties. Recommendation

18

We recommend implementing step by step approach to enter data. Upon clicking “Save and Continue” or “Skip for now” button, the app would automatically move on to the next step of entering data. This method would simplify the user flow. Implement save buttons and feedbacks in all relevant pages

Source or Error In a few areas of the application, participants mentioned wanting more feedback. P4 said "I wish I could see save button. There is no confirmation or anything." P1 said "I think when I hit enter data it won't save it." Priority This has priority of 3. Although lack of system feedback and save buttons did not cause additional errors to tasks, the recommendation has system wide implications. Recommendation We recommend implementing save buttons in all relevant pages such as entering data and changing reminders. We also recommend implementing feedback after saving the changes.

19

Differentiate global navigation color from background

Source or Error We started first task (change reminder) from settings page and 2 out of 4 participants did not know where to start in the app. They both navigated settings page for a while and mentioned “not really sure where to start.” P1 took a wrong path failed the task. P3 noticed the reminder icon after a while and after making few mistakes. Priority This has priority of 3. We gave low priority as we observed that participants learned where things are in the app as we progressed with the test. However, we see that it could be a problem for new users and it could lower user satisfaction as we observed in our test. Recommendation We recommend making the global navigation color different from background color to make it more noticeable. Conclusion

20

In general, users found the overall use of AsthmaMD to be easy to use and included features people with asthma would want to use. Though some with mild asthma felt it they would not use because they don’t track their symptoms. In this test, we did notice a few areas that would warrant additional testing, such as the use of labeling for Well, Worse, and Critical as a means to allow users to categorize their symptom severity. This has system-wide usage. We also realize the limitations of having only mild asthma sufferers tested, and would want to conduct additional tests with patients with more severe asthma. Overall, we recommend implementing consistent placement of save, back and cancel buttons across the app to improve navigation. We also recommend giving users more feedback when entering and saving changes. In addition, we believe simplifying navigation paths as much as possible will improve usability.

21

Appendix Collected Data

Warm up questions

Question 1 2 3 4 *

Do you have asthma? Yes Yes Yes No

How severe are your asthma symptoms?

Mild 1 Not very Very mild 4 No

How often do you visit the doctor because of your asthma symptoms?

Just regular physical

Annual check up

Regular check ups

No

Do you have any obstacles in managing your symptoms? What are those?

Exercise causes it 2

Not really no

No No

How do you currently manage your asthma?

Aware of triggers and keeps inhalers nearby. 3

Rescue inhaler

Albuterol inhaler

No

Do you use AsthmaMD app? No no No No

Do you use any apps to manage your asthma? What you like or dislike about this app?

No no No No

Do you use a smartphone? What kind do you use?

Yes, iphone 6s

Yes Yes, Android Yes

Quotes

1. Bad as a child until 14-15 - didn't have again until 47. Regularly, never full blown, When I have an attack - .5 -1 hour to release lungs without inhaler. Just use inhaler if i need to.

2. Embarrassing when at a gym and void it. Put inhaler in ziplock and put on deck by pool. 3. Humility, exercise, house cleaning fumes. 4. Exercise induced, use inhaler infrequently

*Work around questions about logging habits

TASK 1: Change medication reminder

You just had a doctor visit and the doctor suggests you to take Alvesco at 10pm everyday, once a day, instead of 10am. Use AsthmaMD app to change the medication reminder settings. Mistakes Count

1 2 3 4

Number of clicks on incorrect menu item or button

1 2

22

Does not change “once a day” label on Reminder screen for Alvesco medication

x 4 tries to find

Does not change time for reminder to 10:00 am

Does not click Save to complete tasks

Success | Partial Success | Failure Partial Success Success Failure Task paths (review of reflector video and observation notes)

● (no video - observation is missing some pieces due to not seeing his actions) Start on Setting pages, scrolls around a bit to find reminders, doesn't find it there. See bottom taskbar and clicks on reminders. Finds Alvesco, questions where to change time, finds it and changes it, saves. Done.

● Start on Settings page, scrolls, and clicks on Save (syncs). Found reminders, changes frequency. Clicks on time, uses time wheel to change time, Saves. Goes back to Reminder screen. Done in less than 45 seconds, was very quick.

● Started on Settings page, scrolls through page and clicks on options there. Clicks on Reminders. Clicks to set time with wheel, Saves. Four tries to set once a day on the time screen, but found and changed. Done.

● no video, no notes - hangouts cut out. Observation Quotes

1. "I don't see Alvesco on here, let's see if I can add something" "Where's alvesco? oh, there's the control medicine. Okay, so here is alvesco, I didn't know that was there. So, I took take that out. Frequency is here, one per day. I don't see a time to take it. So….. I still don't find a place to change, oh there is it." "I think that's that" -(Participant clicked on a wrong path “Edit Action Plan” and followed the step by step set up - he is referring to rescue medication list which does not have Alvesco”

2. "Done" 3. "ummm, hm" "Not really sure where to start. Ummm." "On at the bottom - reminders." "at first the calendar

setting, I don't see a time though." 4. "So just by looking at the screen I would go to reminders" "I am not sure I should switch to once a day, I'm

guessing I should change to once a day from as prescribed" "I would change to 10:00pm" "My natural instinct is to go to the right side to save, but that's where the cancel button is, but I did see the save button is over here" "I think I'm doneI "I wish I could see save button. There is no confirmation or anything"

Task 1 Attitude/Emotion - this was adjusted for the report to be 4 categories



Participant Reactions

Confident, Neutral "This was easy" "I was able to do that" “I am done with the task”

Neutral, smiling, confident 1 - 1x quote 4 - 1x quote 2 - 1x quote Most facial expressions were neutral unless noted

Unsure, Confused “I think I did it right” “kind of “ “Am I done?”

Frowning, tilt head to the side (puffing cheeks, squint eyes, questioning looks)

1 - 2x facial 1 - 1x quote 3 - 2x facial 4 - 1x quote 2 - 1x facial


“I was confused” “I was not sure” "I don't know what to do" “I was frustrated”

Angry, shaking their head, sighing, unsure

1 - 4x quotes 3 - 1x quotes 4 - 1x quote

23

"I can't figure this out"

Task 1 Post task questionnaire Question 1 2 3 4

I was able to predict where in the app to start this task.

3 - Started in the setting screen, but it wasn't there. I went to main [medication] screen. But I didn't see the bottom links, with reminders.

5 4 - Instinct was to look at the top, more used to menus at the top. I did see the bottom task bar.

5 - Saw reminder section right away

I could easily recognize the app recorded the information I entered as I expected.

4 - Once I got there a 4, but earlier [in settings screen] I thought it would be there as well.

5 5 - That part is clear, that was pretty easy.

5 - He can see right there 10pm, I wish there was confirmation

Icons and labels guided me efficiently to complete the task.

3 - Partly where I started, but I know for me I see things top down, so I would have noticed the task bar up there, but not so much at the bottom.

5 - Yes, it was easy, they were yeah.

4 - For the most part yea, though I did see the bottom task bar.

4 - bc save button was not clear

Task 1 Common Issues

● 2 users out of 4 didn't see the task bar at the bottom of the screen to navigation from the setting screen to the reminder screen. Users scrolled on settings screen for a bit without finding how to go to reminders.

● 2 of 4 users had issues with changing the frequency, one tried 4 times to find how to do it (looking in the time area), and one skipped this step of the task.

● 1 user did this very fast and completed without mistakes. Task 1 Recommendations

● Improve the visibility of the task bar. ● Improve the layout of the frequency selection.

TASK 2: Log an asthma attack

Imagine you just had an asthma attack after you went for a long run. You took both rescue and control medications but your symptom is worse than usual with frequent coughs and peak flow at 400 l/min. Now you want to record information related to this asthma attack to help you keep track of asthma symptoms over time. Open AsthmaMD to log this asthma attack. Mistakes Count

1 2 3 4


2 2- looked around app

Does not enter peak flow or inputs wrong

24

peak flow

Does not enter symptoms or inputs wrong symptom

Commission of weather

Does not enter trigger or inputs wrong trigger

x

Does not enter medications

Does not click Save to complete task

Success | Partial Success | Failure Partial Success Partial Success Partial Success Success Task paths (review of reflector video and observation notes)

1. (some video missing) Started on Reminders page, Clicked on Enter Data, clicked on dairy-chart, scrolled through it, went back to Enter Data. Clicked on time, went back (enter data button). Clicked on Peak Flow, changed it, was able to use the wheel, went back (enter data button). Clicked on symptoms, scrolled through the list, checked Cough, scrolled through list for 37+ seconds, went back (enter data button). Clicked on symptoms, and scrolled through list for a second time, went back (enter data button). Clicked on triggers, scrolled through list, selected Exercise and Weather (cold air), went back (enter data button). Clicked on Medication, stayed there for several seconds, selected Brethine and Alvesco, scrolled a bit, went back (enter data button). On main screen for 15 second before clicking Save. Dialog showed "Start action plan?", clicked yes. Was taken to action plan screen - green zone.

2. Started on Reminders page. Clicked on Enter Data. Clicked on dairy chart scrolled through. Went to Enter Data again, stayed on screen a bit. Clicked back and forth on enter data options, scrolled each screen. Clicked on symptoms, checked cough, scrolled screen a bit, went back (enter data button). Click on peak flow, entered peak flow with wheel, went back (enter data button). Clicked on Triggers, scrolled, check exercise, went back (enter data button). Clicked meds, scrolled, looked for control and rescue, selected Alvesco and Brethine, went back (enter data button). Click Save. Dialog showed "Start action plan?", clicked No. Done.

3. Started on Reminders. Clicks on Enter Data. Clicks on Symptoms, scrolls through page, selects Coughing, scrolls through page a bit, went back (enter data button). Clicks on peak flow, enters value on wheel, went back (enter data button). Clicks medication, selects Brethine and Alvesco, went back (enter data button). Click Save. Dialog showed "Start action plan?". Done.

4. Started on Reminders page. Clicks on Enter Data. Clicks on peak flow, changes it with wheel, went back (enter data button). Clicked on symptoms, scrolls through, selected Coughs, scrolls through, went back (enter data button). Clicks on Triggers, scrolls through, selects Exercise, went back (enter data button). Clicks Medication, pauses a bit, selects Brethine and Alvesco, went back (enter data button). Click Save. Dialog showed "Start action plan?". Done.

Observation Quotes

1. "Okay I see, under data, change the time, peak flow" "I think when I hit enter data it won't save it" "So worse is cough" "so that's good" "It doesn't say an actual attack [when on symptoms] I would look for more about that." "It doesn't say that I took or give option that I took the control as attack or as normal medication." "Alright, now I am done after saving"

2. "Here's some buttons" "I'm going through pages to see what I can do." "Ah, that's what it is" "I am trying to figure out where to log this is running."

3. "I see enter data" "hmm" "I feel like that peak flow information is given, I feel like I should be able to enter it somewhere. But I don't see anywhere where is makes sense." "Oh here we go backing up." That makes sense"

4. "So, i would say go to enter data. And I'm not sure this is where I would enter an attack, I would change date if it needs to be changed, I click on peak flow and change it to 400, i guess I click on enter data. I click on symptoms - frequency coughs" "I don't see anything else, click on triggers, I don't' see… oh exercise, back. Medication, I guess I click on" " the rescue and control medications. Back and now I click save." " Start asthma action plan? I'm not sure what that means?"

25

Attitude/Emotion - this was adjusted for the report to be 4 categories





Neutral, smiling, confident 1 - 3x quote 3 - 2x quotes 4 - 1x quote 2 - 1x quote Most facial expressions were neutral unless noted


Frowning, tilt head to the side (puffing cheeks, squint eyes, questioning look)

1 - 1x quote 2 - 2x quote 4 - 3x quote 3 - 1x facial 3 - 1x quote 4 - 1x facial


“I was confused” “I was not sure” "I don't know what to do" “I was frustrated” "I can't figure this out"


4 - 1x quote 2 - 2x quote

Post task questionnaire Question 1 2 3 4


5 - Enter data was clear

4 - Because I hadn't seen it before I needed to look around, assumed enter data was the right place, but I want to get familiar with it.

5 - It was easy to figure out where to go.

4 - I think it was sort of intuitive to enter data, i would have anticipated something that said attack to trigger me to enter data for an attack.


4 - I give that a four because I wasn't sure the enter data was a back button and if it would save.

5 - That was true. 5 - That was very clear with the buttons

5 - Yeah that's a 5


4.5 - same reason as above

4 - Mostly because you need to know what things mean, to get familiar with it.

5 - Everything was clearly labeled so it was pretty clear.

4 - It you are going to enter multiple points, I would think there would be a next better to go all the way through to save.

Common Issues

● 3 out of 4 users were unsure if enter data button to go back to main screen would save the data, they were unsure what the button did or if it was a button.

● 2 out of 4 users wanted to enter data about if this was to record normal medication or if this record was related to an attack. (possibly showing us the user doesn't understand the well, worse, and critical labeling as it related to severity of symptoms).

26

● 1 of 4 users mentioned being curious about where things were in the app, wanted to review the app to see where things were before completing tasks.

Recommendations

● Improve the visibility of the enter data button to make it more clear what it does. ● Well, Worse, and Critical was not sufficient labeling to help users to understand how the severity of their log

is recorded.

TASK 3: Share asthma record with physician

Imagine you have a doctor visit coming up in two days to talk about changes in the frequency of your asthma attacks. You want to send data you have logged in the past on the app to discuss with Dr. Grey. Use AsthmaMD app to send your report to Dr. Grey. Mistakes Count

1 2 3 4


Does not uncheck other contact list x

Does not click Send Report

Success | Partial Success | Failure Partial success Success Success Success Task paths (review of reflector video and observation notes)

1. Started on Action Plan - Green zone page. Clicks on Diary-chart, scrolls a bit. Clicks Send report buttons. Goes to Send Report screen. Clicks Send Report button, Dialog opens, clicks "okay", report sends. Report sent to 3 recipients. Done.

2. Started on Enter Data page. Clicked on Diary Chart, scrolled page, Clicked Send Report button, goes to Send Report page. Unchecked all but Dr Grey. Clicks Send Report button, Dialog opens, clicks "okay", report sends. Report sent to 1 recipients. Done.

3. Started on Enter Data. Clicked Diary Chart, scrolled screen. Clicked on Send Report button, goes to Send Report screen. Unchecked all but Dr Grey. Clicks Send Report button, Dialog opens, clicks "okay", report sends. Report sent to 1 recipients. Done.

4. Started on Enter Data page. Clicks Diary-chart. CLicks Send Report Button, Unchecked all but Dr Grey. Clicks Send Report button, Dialog opens, clicks "okay", report sends. Report sent to 1 recipients. Done.

Observation Quotes

1. "So first thought was diary chart because it has my records." And there's a send report button right here, it's highlighted, so I'm going to click, it's get the names to notify, they are already checked, so I am going to send report."

2. "Seems like dairy chart is not for recording as a database. I am guessing I should send to Dr Grey. Yep." 3. "my first instinct is to look for the data, I figured it might be going from there. But Chart, I see what to do." 4. "I would probably go to diary-charts, I am going to send to dr grey and uncheck the other names in there

and hit send report and I got a confirmation." Attitude/Emotion - this was adjusted for the report to be 4 categories




27


Neutral, smiling, confident 1 - 3x quote 1 - 1x facial 2 - 2x quote 3 - 1x quote Most facial expressions were neutral unless noted


Frowning, tilt head to the side (puffing checks, squint eyes, questioning look)

4 - 1x quote 4 - 1x facial


“I was confused” “I was not sure” "I don't know what to do" “I was frustrated” "I can't figure this out"


Post task questionnaire Question 1 2 3 4


5 - Nothing else sounded like it, it was a guess, if i looked elsewhere I think i would have been frustrated.

5 - Once you know where to look, you send from where it's stored from.

5 - Made sense, based on the other menus to go there.

5 - It was in charts


5 - It was a guess that the function would there.

5 - Yes, I could see it was sending it

5 5 - Confirmation was sent


4 - Yes, but there was no text to actually know what it would do.

5 - As far as it goes, send report was a clear indicator

5 - Buttons were clear.

5 - It was clear

Were you aware of what information was being sent in the report .

"It is sent as a text file?"

I don't know how the information is received, but based on the selection up here, I think it's sending those. (pointing to All)

Can see on the mainscreen what was being sent, so that was clear.

Oh, I didn't send report. Yeah the information that is here (main screen)

Common Issues

● Everyone likes databases! ● 4 out of 4 users believed that what they saw on the main screen was what was being sent. ● 2 out of 4 users mentioned they were unsure of how the data was sent.

Recommendations

● At this point users were more comfortable with the app and the learning effect had taken place.

28

● Make it more clear what exact information is being sent.

Post Study Satisfaction Questionnaire

Please rate the following categories by marking the box based on your overall experience.



agree



2 4 1 3


2 3 1 4


2 1 3 4


2 3

1 4

Post-study questions In the post study questionnaire, you mentioned the app fits/does not fit your needs? Can you explain further?

1. I like to be able to see what I had and what caused it. If I had increase in attacks or symptoms it would be nice to know what that means. If I wake up in the middle of the night, I would like to know if frequency has changed.

2. I wouldn't use frequency because I don't need too much. But would suggest to others. I only use rescue once a month or so, When I was younger I took rescue 2-3 day, and that would be more useful. I don't know if you can create other profiles. But not useful for me.

3. Can be useful, But i don't track but can be helpful, Would use if I had symptoms, Good to see doctor, really really helpful

4. I think it is nice to have the app track symptoms and medications

Did the app match your expectations what an asthma management application should do? 1. Yes. but what I was just talking about I would like to know changes, strong attacks versus mild,

something to see that - Like fitbit has graphs I can see. The data would be there. 2. Yes. 3. Yeah, it was pretty clear, a lot of features, it was intuitive to enter data. 4. I think so

What did you like about Asthma MD? Were these easy to understand?

1. It was easy to understand. I take care of a lot of seniors from families and I think it would be more difficult for them. The wizards were good. But I had no idea there was a different screen for control medication. The six steps would be good.

2. Usability is simple, which is great. I liked you can send before a doctor visit. 3. Very beginning unclear, a lot of acronyms. Was confusing. Labels at the bottom were not

intuitive, but I was able to understand with context.

29

4. Yeah i think so, i like data and being able to run reports and send to others.

What did you not like about AsthmaMD? Were these difficult to understand? 1. I think if the report is sent through email it should go to you also, because it records and exports

it's good. We didn't do every screen but I think symptoms page adding a class for type of attack (severity) a check box would be helpful.

2. Not really anything. 3. Very beginning unclear, a lot of acronyms. Was confusing. Labels at the bottom were not intuitive,

but I was able to understand with context. 4. There were a couple scenarios I questioned what buttons to click.

If there was something you could improve to make the application better for your needs, what would that be?

1. Designing for a senior, language can overwhelm them. They aren't as used to smartphone and touch, and loss of sensitivity in fingers, tough didn't translate.

2. The only thing is having different profiles for a mom who is managing their children's asthma. 3. Is there any visualization with the report? Highlight the days. [moderator shows need to tilt]

ohh…. That is helpful. 4. Nothing off top of my head.

What general comments /questions do you have about AsthmaMD?

1. Is it free, and available? 2. No 3. No 4. No

What is your highest level of education?

1. Bachelors 2. Masters 3. Graduate school 4. College degree

What is your occupation?

1. In between jobs. IT Consultant, mergers and acquisitions. state farm, did usability testing with it. 2. Administrative assistant 3. Campaign worker - political strategist 4. Training director

How old are you?

1. 50 - can't share that 2. 27 3. 28 4. 37

Do you have any questions for us, or any general comments?

● No. good luck. ● No ● I don't think so. ● No

30

End of Collected Data

Adult Consent to Participate in Research

Team: Sup Suh, Ann Junker Institution: College of Computing and Digital Media – Human-Computer Interaction Master’s Program, DePaul University What is the purpose of this research? The goal of our research is to assess usability issues with AsthmaMD mobile application. We are not testing you. We are testing the product. Your help during this test will aid us in making recommendations to make the app easier to use. This research is being supervised by our faculty advisor, Enid Montague. We hope to recruit 5 people for the interview in the research. Why are you being asked to be in the research? You are invited to participate in this study because you have Asthma. You must be age 18 or older, own or have used a mobile device to be in this study and speak fluent English. What is involved in the research study? If you agree to be in this study, the research will involve conducting a 15 minutes usability test and 15 minutes of pre & post testing. The facilitator will ask you to perform a series of tasks with the product and we will ask you follow up questions after each task. This will help us understand how easy it is to use this product. Consent for recording We will be recording the audio recording of the test, video recording of mobile screen and depending on your consent, video recording of you performing the tasks. Please check-mark an option below and sign your name and date. [ ] I only allow the study to be audio/voice recorded for the purposes of gathering data. My information is shared. The research may not use my real name or disclose any information that may reveal my identity. [ ] I allow the study to be audio/voice/video recorded. I allow the study to use video footage of me to demonstrate the test results and may use footage for portfolio website. You may ask to take down the video at any time after the study if you do not wish your footage to be included. Name: __________________ Date: __________________ Signature: ___________________ Are there any risks involved in participating in this study? Being in this study does not involve any risks other than what you would encounter in daily life. You may feel uncomfortable or embarrassed about performing certain tasks. Are there any benefits to participating in this study? You will not personally benefit by participating in this study. Your insights will allow us to better understand the usability issues of using AsthmaMD app. Are there any costs to me for being in the research? There is no cost to you for being in this research study. Can you decide not to participate?

31

Your participation is voluntary, which means you can choose not to participate. There will be no negative consequences, penalties, or loss of benefits if you decide not to participate or change your mind later and withdraw from the research after you begin participating. Who w ill see my study information and how w ill the confidentiality of the information collected for the research be protected? If you have agreed on video recording consent, your name and video footage will be disclosed. If you have not agreed on video recording consent, we will keep your data confidential. We will not include your name or any information that will directly identify you. We will make every effort to prevent anyone who is not on the research team from knowing that you gave us information, or what that information is. All information will be coded and names will be altered. This is meant to keep all participant information confidential. What if new information is learned that might affect my decision to be in the study? If you no longer wish to participate, you have the option to withdraw from this interview at any time. You can also request the moderator to refrain from documenting anything you do not wish to have recorded. Who should be contacted for more information about the research? Before you decide whether to accept this invitation to take part in the study, please ask any questions that might come to mind now. Later, if you have questions, suggestions, concerns, or complaints about the study or you want to get additional information or provide input about this research, you can contact the researchers: Sup Suh ([email protected]), Ann Junker ([email protected]) Consent signature to participate I have received a copy of this consent form. I agree to participate.

Participant’s Signature: ___________________________________ Date: ________________ Facilitator’s Signature: ____________________________________ Date: ________________

End of Consent Form Screener questions

Survey created on google form Asthma Management Application Invitation As a graduate student project team conducting a usability study, we are looking for people who are interested in providing feedback and improving the design of an asthma management mobile application.

Question Purpose

Do you have asthma? ● Yes - continue with survey ● No - thank user and end survey

Primary screener question

How often do you have asthma symptoms? ● Fewer than 3 times a month ● 1-5 days a week, but not all day ● Once a day, but not all day ● Throughout the day most days

To group participant into one of the characteristic groups

mailto:[email protected]

mailto:[email protected]

https://docs.google.com/forms/d/1K5fptBVWDZJonH5uvXQ65Fwm63TSU4Y9Mm2F6hATUZE/edit

32

Do you currently use a smartphone? ● Yes ● No

To ascertain their experience with smartphones

Are you 18 years of age or over? ● Yes ● No

To meet test age restriction

Do you accept to have the study session recorded? We record the steps you take, and keep personal information private.

● Yes ● No

To meet test monitoring requirements

Full Name ● Single input text field

Gather contact information for follow up communication

Email Address ● Single input text field

Gather contact information for follow up communication

End of Screener Questions Moderator Guide

Check List

Moderator 1. Consent Form x 5 copies 2. Task Sheets with Survey x 5 copies 3. Moderator Guide

iPhone with 5 accounts set up with participant profile, medications, doctor’s email and pre-entered symptoms. Read the steps below.

Notetaker 4. Notetaker Log Sheet x 5 copies

Participant Consent Form, Task Sheets with Survey (Given by moderator)

iPhone (Given by moderator)

Pre-test set up

1. Moderator should have this document and read outloud 2. Observer should have log sheet to take notes. 3. Make sure AsthmaMD application is set up

a. New user account should be created before each test so that it does not carry old data. b. Account name:

i. Participant 1 ii. Participant 2 iii. Participant 3 iv. etc.

c. Doctor’s name and email is already entered: i. Doctor name: Doctor Grey ii. Email: [email protected]

d. Medications are already entered i. Control: Alvesco 80mcg

33

ii. Rescue: Brethine 2.5mg iii. Under Reminders - have As Prescribed selected for each medication.

e. Entered Data already entered i. Entry One

1. Date: February 6, 2018 8:15 PM 2. Symptoms: Wheeze, Shortness of breath 3. Medication: Brethine

ii. Entry Two 1. Date: February 7, 2018 7:00 AM 2. Symptoms: Coughing, wheezing 3. Medication: Alvesco

4. Another phone for video/audio recording is ready and reflector is ready. 5. Give participant the document, a pen and iphone with AsthmaMD

Possible probing questions during testing

These are examples that can be used with the moderator finds the participant is confused. ● I noticed you hesitated on this piece, was there something that was confusing? ● Is there something that could be more helpful at this step? ● It seems you are getting frustrated, this step is crucial to the app, please try the task more? ● It seems you are getting frustrated, others have too, please try the task more? ● I can't answer that question, but I can ask you where do you think it should be? How do you think it should

behave?

Start Usability Test (Introduction & Consent Form)

I am _____ and will be moderating the usability test today and _____ will be recording the data and taking notes. Thank you for joining the usability session today for the AsthmaMD. This study will help us understand the how users, like yourself, interact with the application, and show us where improvements in the design and usability are needed. We will ask you to complete three tasks and to answer few questions about your experience with the tasks. We will start with the consent form. Go over the consent form. Do you have any questions about the consent form and the study?

Warm up questions

We will start by asking a few questions to help us understand how you currently manage your asthma symptoms. Ask follow ups based on participant's responses, if necessary.

● Do you have asthma? ● How severe are your asthma symptoms? ● How often do you visit the doctor because of your asthma symptoms? ● Do you have any obstacles in managing your symptoms? What are those? ● How do you currently manage your asthma? ● Do you use AsthmaMD app? ● Do you use any apps to manage your asthma? What you like or dislike about this app? ● Do you use a smartphone? What kind do you use?

34

Okay, we will begin to record the session.

Usability Testing

Start Recording.Start Reflector (Scroll Down Iphone screen to select “Screen Mirroring, Get 4 digit code to enter on screen to mirror screen). Observer should have blank log sheet ready. Now, we will ask you to perform three tasks. As a reminder, we are not testing you, but rather testing the product. There are no wrong answers. Please think out loud while performing each task. Thinking out loud helps us understand what you find interesting or confusing, and also any comments you may have about the application. All the feedback you provide will help understand how to improve AsthmaMD. Any questions before we begin? Great, now we will begin the usability testing.

Task 1: Change medication reminders

Application should start on the Settings screen Moderator reads task out loud and hands participant scenario sheet from participant guide. You just had a doctor visit and the doctor suggests you to take Alvesco at 10pm everyday, once a day, instead of 10am. Use AsthmaMD app to change the medication reminder settings. Please begin the task and let us know you are done. Wait for task to be completed. Now, I have a few questions about your experience with this task. Hand participant the questionnaire for this task - questionnaire available in participant guide. Ask questions outloud to the participant. Thank you. Now we will start the second task.

Task 2: Log an asthma attack

Application should start on the last screen user was on. Moderator reads task out loud and hands participant scenario sheet from participant guide. Imagine you just had an asthma attack after you went for a long run. You took both rescue and control medications but your symptom is worse than usual with frequent coughs and peak flow at 400 l/min. Now you want to record information related to this asthma attack to help you keep track of asthma symptoms over time. Open AsthmaMD to log this asthma attack. Make sure you fill in all the data. Please let us know you are done with the task. Wait for task to be completed. Now, I have a few questions about your experience with this task.

35

Hand participant the questionnaire for this task - questionnaire available in participant guide. Ask questions outloud to the participant. Thank you. Now we will start the third task.

Task 3: Share asthma record with physician

Application should start on the last screen user was on. Moderator reads task out loud and hands participant scenario sheet from participant guide. Imagine you have a doctor visit coming up in two days to talk about changes in the frequency of your asthma attacks. You want to send data you have logged in the past on the app to discuss with Dr. Grey. Use AsthmaMD app to send your report to Dr. Grey. Please let us know you are done with the task. Wait for task to be completed. Now, I have a few questions about your experience with this task. Hand participant the questionnaire for this task - questionnaire available in participant guide. Ask questions outloud to the participant. Thank you. That was the last task.

Wrap-up Questionnaire

Hand participant post-study questionnaire - available in participant guide. Now we will ask you to fill out post-study questionnaire. Please let us know you are done. Wait Thank you. Now we would like to ask a few follow up questions to sum up your experience with the application

Wrap-up Questions

Ask follow ups based on participant's responses.

1. In the post study questionnaire you mentioned the app fits/does not fit your needs? Can you explain further?

3. Did the app match your expectations what an asthma management application should do? 4. What did you like about Asthma MD? Were these easy to understand? 5. What did you not like about AsthmaMD? Were these difficult to understand? 6. If there was something you could improve to make the application better for your needs, what would that

be? 7. What general comments /questions do you have about AsthmaMD? 8. What is your highest level of education? 9. What is your occupation? 10. How old are you? 11. Do you have any questions for us, or any general comments?

36

Thank you, this concludes our session today.

End of Moderator Guide Participant Guide

Participant Guide

Pages 17 - 23 - each page handed separately to participant

Task 1: Change medication reminder

You just had a doctor visit and the doctor suggests you to take Alvesco at 10pm everyday, once a day, instead of 10am. Use AsthmaMD app to change the medication reminder settings. Please let us know you are done with the task.

Task 1 questions: Change medication reminder Now that you’ve completed the task, with the scenario in mind, from a scale from one to five (1 very difficult , 5 very easy) how would you rate the following statements:

1. I was able to predict where in the app to start this task.

Score: _____ Why?______________________________________________________

2. I could easily recognize the app recorded the information I entered as I expected.

Score: _____ Why?______________________________________________________

3. Icons and labels guided me efficiently to complete the task.

Score: _____ Why?______________________________________________________

Task 2: Log an asthma attack Imagine you just had an asthma attack after you went for a long run. You took both rescue and control medications but your symptom is worse than usual with frequent coughs and peak flow at 400 l/min. Now you want to record information related to this asthma attack to help you keep track of asthma symptoms over time. Open AsthmaMD to log this asthma attack. Please let us know you are done with the task.

37

Task 2 questions: Log an asthma attack

Now that you’ve completed the task, with the scenario in mind, from a scale from one to five (1 very difficult , 5 very easy) how would you rate the following statements:


Score: _____ Why?______________________________________________________


Score: _____ Why?______________________________________________________


Score: _____ Why?______________________________________________________

Task 3: Share asthma record with physician Imagine you have a doctor visit coming up in two days to talk about changes in the frequency of your asthma attacks. You want to send data you have logged in the past on the app to discuss with Dr. Grey. Use AsthmaMD app to send your report to Dr. Grey. Please let us know you are done with the task.

Task 3 questions: Share asthma record with physician

Now that you’ve completed the task, with the scenario in mind, from a scale from one to five (1 very difficult , 5 very easy) how would you rate the following statements:


Score: _____ Why?______________________________________________________


Score: _____ Why?______________________________________________________


Score: _____ Why?______________________________________________________

4. Were you aware of what information was being sent in the report .

Why?_____________________________________________________

38

Post Study Satisfaction Questionnaire

Please rate the following categories by marking the box based on your overall experience.



agree






When you are done, please hand this questionnaire back to the moderator. After you have completed this questionnaire, the moderator will ask a few follow up questions.

End of Participant Guide

Observation Guide Participant code name: Date/Time:

● Do you have asthma?

● How severe are your asthma symptoms?

● How often do you visit the doctor because of your asthma symptoms?

● Do you have any obstacles in managing your symptoms? What are those?

● How do you currently manage your asthma?

● Do you use AsthmaMD app?

● Do you use any apps to manage your asthma? What you like or dislike about this app?

● Do you use a smartphone? What kind do you use?

Participant code name: Date/Time:

39

TASK 1: Change medication reminder

Mistakes Count


Does not change “once a day” label on Reminder screen for Alvesco medication

Does not change time for reminder to 10:00 am

Does not click Save to complete tasks

Think aloud Observer Notes Attitude and emotions, think aloud quotes, feedback

Able to complete task Success | Partial Success | Failure

Post task questionnaire


Score: _____ Why?_____________________________________________________


Score: _____ Why?_____________________________________________________


Score: _____ Why?_____________________________________________________


Task 2: Log an asthma attack

Mistakes Count


Does not enter peak flow or inputs wrong peak flow

Does not enter symptoms or inputs wrong symptom

Does not enter trigger or inputs wrong trigger

Does not enter medications

40

Does not click Save to complete task

Observer Notes Attitude and emotions, think aloud quotes, feedback




Score: _____ Why?_____________________________________________________


Score: _____ Why?_____________________________________________________


Score: _____ Why?_____________________________________________________


TASK 3: Share asthma record with physician

Mistakes Count


Does not uncheck other contact list

Does not click Send Report

Observer Notes Attitude and emotions, think aloud quotes, feedback




41

Score: _____ Why?_____________________________________________________


Score: _____ Why?_____________________________________________________


Score: _____ Why?_____________________________________________________

4. Were you aware of what information was being sent in the report .

Why?_______________________________________________________________


Post Testing Questions

1. In the post study questionnaire you mentioned the app fits/does not fit your needs? Can you explain

further?

2. Did the app match your expectations what an asthma management application should do?

3. What did you like about Asthma MD? Were these easy to understand?

4. What did you not like about AsthmaMD? Were these difficult to understand?

5. If there was something you could improve to make the application better for your needs, what would that be?

6. What general comments /questions do you have about AsthmaMD?

7. What is your highest level of education?

8. What is your occupation?

9. How old are you?

10. Do you have any questions for us, or any general comments?

AsthmaMD Usability Test Report

Documents

Transcript of AsthmaMD Usability Test Report