Designing for diversity: A tablet-based ePRO system to ... · Designing for diversity: A...

Designing for diversity: A

tablet-based ePRO system to

assist stroke rehabilitation

Karolina Jonzén

June 5, 2017Master's Thesis in Interaction Technology and Design, 30 credits

Supervisor at CS-UmU: Lars-Erik JanlertExternal supervisor at MT-FoU: Fredrik Öhberg

Examiner: Anders Broberg

Umeå University

Department of Computing Science

SE-901 87 UMEÅSWEDEN

Abstract

In this thesis, a web- and tablet-based ePRO system to assist stroke rehabilitation was de-veloped in an iterative and user-centered design process involving elderly and people withpersonal and professional experience of stroke. A limited part of a paper-based PRO ques-tionnaire was transitioned into the developed electronic format and evaluated on �ve strokepatients using a qualitative usability testing approach, including observation and feedbackthrough discussion and a post-test questionnaire. The results indicated that a tablet-basedePRO questionnaire could be both usable and applicable in this target group, with a ma-jority of the patients preferring it over the paper-based questionnaire. Furthermore, the�ndings suggest tapping problems and cognitive overload to be speci�cally considered whendesigning ePROs for this target group, which may also bene�t from a certain level of adap-tivity to achieve personalized user interfaces and increased accessibility.

Contents

1 Introduction 1

1.1 Scope of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1.1 The Rehab-Compass . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Problem Description 5

2.1 Goals and Purposes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Restrictions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2.3 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Methods 7

3.1 Pre-study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.2 Design phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.2.1 User testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.4.1 Study design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.4.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.4.3 Test setting, user task and procedure . . . . . . . . . . . . . . . . . . . 9

3.5 Usability questionnaires . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.6 Equipment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4 Background 11

4.1 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.2 Designing ePROs for stroke patients . . . . . . . . . . . . . . . . . . . . . . . 12

4.2.1 Stroke and its implications . . . . . . . . . . . . . . . . . . . . . . . . 13

4.2.2 Design guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5 Designing for diversity 15

5.1 Adaptive user interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.1.1 Achieving personalization . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.1.2 Techniques: Adaptive presentation and navigation . . . . . . . . . . . 16

iii

iv CONTENTS

5.1.3 Potential and challenges . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5.2 Examples: Using adaptivity to increase accessibility . . . . . . . . . . . . . . 17

5.2.1 The SUPPLE system . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

5.2.2 MyUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

6 Development process 19

6.1 Choosing the questionnaire format . . . . . . . . . . . . . . . . . . . . . . . . 19

6.1.1 Item presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

6.1.2 Response options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

6.2 Iteration 1: Low-�delity prototypes . . . . . . . . . . . . . . . . . . . . . . . . 21

6.2.1 Prototype 1 � One item per screen . . . . . . . . . . . . . . . . . . . . 21

6.2.2 Prototype 2 � Multiple items per screen . . . . . . . . . . . . . . . . . 22

6.2.3 User tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

6.3 Iteration 2: Medium-�delity prototypes . . . . . . . . . . . . . . . . . . . . . 23

6.3.1 User tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

6.3.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

6.4 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

6.4.1 The database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

6.4.2 The questionnaire framework . . . . . . . . . . . . . . . . . . . . . . . 28

6.4.3 The Rehab-Compass chart . . . . . . . . . . . . . . . . . . . . . . . . . 29

6.5 Ful�llment of requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

7 Results 31

7.1 Final prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

7.2 User evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

7.3 Feedback, comments and observations . . . . . . . . . . . . . . . . . . . . . . 35

8 Discussion 37

8.1 Potential and usability of the ePRO questionnaire . . . . . . . . . . . . . . . 37

8.2 Increasing accessibility of the system . . . . . . . . . . . . . . . . . . . . . . . 38

8.2.1 Value of adding adaptivity . . . . . . . . . . . . . . . . . . . . . . . . . 38

8.3 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

9 Conclusions 41

9.1 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

10 Acknowledgments 43

References 45

List of Figures

1.1 Prototype of the Rehab-Compass chart created in Excel. . . . . . . . . . . . . 2

3.1 Work Breakdown Structure of the project. . . . . . . . . . . . . . . . . . . . . 7

6.1 Question group as presented in the original paper questionnaire. . . . . . . . 19

6.2 The VAS used in the last question of the original paper questionnaire. . . . . 20

6.3 The VAS used in both prototypes. . . . . . . . . . . . . . . . . . . . . . . . . 21

6.4 Prototype presenting one item per screen. . . . . . . . . . . . . . . . . . . . . 21

6.5 Overview presented in the one-item-per-screen-questionnaire. . . . . . . . . . 22

6.6 Prototype presenting multiple items per screen. . . . . . . . . . . . . . . . . . 22

6.7 Two di�erent states of a question screen in the second iteration of the one-

item-per-screen prototype. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

6.8 Second version of the multiple-items-per-screen prototype. . . . . . . . . . . . 24

6.9 A user testing the multiple-items-per-screen prototype. . . . . . . . . . . . . . 26

6.10 Database schema. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

7.1 The introduction screen. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

7.2 A completed Likert-scale question. . . . . . . . . . . . . . . . . . . . . . . . . 32

7.3 A not completed Likert-scale question. . . . . . . . . . . . . . . . . . . . . . . 32

7.4 The overview presented in-between two parts. . . . . . . . . . . . . . . . . . . 33

7.5 A not completed VAS-scale question. . . . . . . . . . . . . . . . . . . . . . . . 33

7.6 The web-based version of the Rehab-Compass chart. . . . . . . . . . . . . . . 34

7.7 Comparison between the tablet- and paper-based version of the questionnaire. 35

v

vi LIST OF FIGURES

List of Tables

4.1 Design guidelines for ePRO questionnaires aimed at stroke patients. . . . . . 14

6.1 Users and their choice of prototype. . . . . . . . . . . . . . . . . . . . . . . . 25

6.2 Average user rating (1=very di�cult; 5=very easy). . . . . . . . . . . . . . . . 25

6.3 Requirements of the �nal prototype and whether they were ful�lled. . . . . . 30

7.1 Users participating in the �nal user test. . . . . . . . . . . . . . . . . . . . . . 34

7.2 Average user rating (1=very di�cult; 5=very easy). . . . . . . . . . . . . . . . 35

vii

viii LIST OF TABLES

Chapter 1

Introduction

Each year, 15 million people worldwide su�er from a stroke � a brain injury where bloodsupply to the brain suddenly gets interrupted, causing brain cells to die from oxygen de-privation [1, 2]. Depending on its type, location and severity, a stroke may result in deathor cause various cognitive and physical problems, such as loss of sensation and di�cultiesspeaking, seeing or walking [3]. In fact, stroke is one of the leading causes of permanentdisability in adults [4], and as the world's elderly population rapidly grows [5], the numberof stroke survivors with long-term impairments is increasing, setting even higher demandson the health care system to treat these individuals in an e�ective way.

Getting the right rehabilitation treatment early on is essential for stroke patients to beable to recover lost functionality and regain independence [6]. Nelson and colleagues [7]state that one of best ways of understanding the e�ciency of a treatment is through the useof patient-reported outcome measures (PROM's), that capture how the patients experiencetheir own health, quality of life and functional status. Furthermore, the authors describehow the development of PROM's has become increasingly important to adopt a patient-centered approach in medical care. However, since the impairments post-stroke can be verycomplex, a�ecting various functions and aspects of well-being, it can be di�cult for cliniciansto get a holistic view of the patients' rehabilitation needs.

With the rise of mobile technologies and e-health, i.e. �the use of information andcommunication technologies (ICT) for health� [8], new possibilities of enhancing the reha-bilitation process have opened up. ICT solutions can reduce the communication gap betweenpatients and clinicians, as well as encourage patients to take a more active role in their owntreatment, without the inconvenience of long wait times. The collection of patient-reportedoutcome (PRO) data is one area in which such digital and automated systems are increasing,through the use of electronic PRO (ePRO) systems, where patients can report their healthstatus through digital questionnaires, either in connection to a clinical visit or from theirhomes over the Internet [9]. However, little seems to be known regarding the use of ePROsystems in stroke rehabilitation.

1.1 Scope of this thesis

This thesis aims to make a contribution to the development of e-health services for strokepatients by designing, implementing and evaluating a tablet-based prototype ePRO systemthat supports both patients and clinicians in the identi�cation of the patient's rehabilitationneeds. The work is part of a research project called the Rehab-Compass, initiated by the

1

2 Chapter 1. Introduction

department of Rehabilitation Medicine at Umeå University, Sweden, that practices researchand education, focusing on neurological diseases and supporting people with disabilities toregain independence and improve quality of life [10]. The project is assisted by BiomedicalEngineering � Research and Development (MT-FoU) � a department at Umeå UniversityHospital that practices research, development and education in close collaboration with thehealth care sector and Umeå University [11].

1.1.1 The Rehab-Compass

In 2015, the department of Rehabilitation Medicine initiated a project with the purpose ofdeveloping a tool that can assist clinicians in recommending rehabilitation treatments forstroke patients, by graphically visualizing a radar chart that shows a broad spectrum of thepatient's experienced rehabilitation status on an outcome scale from 0�100. A stand-aloneversion of the Rehab-Compass chart was created in Excel, see Figure 1.1 below.

Figure 1.1: Prototype of the Rehab-Compass chart created in Excel.

The targeted patient group also includes people having su�ered from a Transient IschemicAttack (TIA), often called a mini-stroke, where symptoms usually disappear within 24 hoursand generally not cause any permanent brain damage, but can be a warning sign prior toan actual stroke [12].

While not yet used in clinical practice, the Rehab-Compass chart was tested in a pilotstudy conducted by Hu et. al [13], where it appeared to be both time-saving and feasible in

1.2. Thesis outline 3

clinical use. Data from 23 patients was collected at �ve- and twelve-month follow-ups usingsix well-validated and reliable paper-based PRO questionnaires regarding di�erent aspectsof functional status and quality of life. The results were then converted into the 0 to 100outcome scale and manually transferred into an Excel �le where the Rehab-Compass chartswere constructed.

1.2 Thesis outline

The following chapters are organized as follows:

� Chapter 2: provides a more detailed description of the goals and purposes of thethesis, as well as restrictions and requirements on the �nal prototype.

� Chapter 3: describes how the work was carried out and the methods used.

� Chapter 4: present background information, including related work and design guide-lines found in the literature.

� Chapter 5: in-depth study on designing for diversity, focusing on adaptive userinterfaces and their application in user groups with special needs.

� Chapter 6: describes the development process leading up to the �nal prototype.

� Chapter 7: presents the resulting prototype and its associated user evaluation.

� Chapter 8: discusses the results and �ndings in relation to previous studies andidenti�es the main limitations of the study.

� Chapter 9: closes the thesis with concluding remarks that summarize the key �ndingsand provides suggestions for future work.

4 Chapter 1. Introduction

Chapter 2

Problem Description

Understanding and meeting the multidimensional needs of stroke patients, and providingthem with direct and accessible feedback regarding their rehabilitation status are importantfactors to a successful treatment. The Rehab-Compass project � generating an easy-to-graspgraphical view representing the patients' own perceptions of their problems � was startedby the department of Rehabilitation Medicine as a mean to achieve this.

While the instrument has been showing promising results as a stand-alone prototype inExcel, the PRO questionnaires are currently paper-based, requiring all questionnaire datato be manually inputted or scanned by clinicians � a time-consuming task, also preventingpatients from getting immediate feedback. To streamline the process, the goal is to havethe Rehab-Compass chart integrated as a part of an e-service, where patients easily cancomplete the questionnaires and access their results.

2.1 Goals and Purposes

The goals of this thesis are:

� To design and implement a tablet-based prototype of an ePRO system that integratesthe whole process from data collection of the patients' rehabilitation status to theresulting view of the Rehab-Compass chart.

� To evaluate the potential and usability of a questionnaire integrated in the developedePRO system, and compare it to the paper-based version.

The intent is that such an e-service could be used by both patients and clinicians tosupport the identi�cation of the patients' rehabilitation needs, and ultimately contribute toa better and more e�cient treatment. The thesis should also provide valuable insights as towhat usability factors that could be especially important to consider when designing ePROsfor stroke patients.

Additionally, the thesis explores modern strategies of designing user interfaces for diverseuser groups by adding adaptivity to the systems. More speci�cally, it seeks to answer thefollowing question:

� How could adaptivity be used to increase accessibility of the ePRO system, withrespect to the diverse physical and cognitive limitations seen in stroke patients?

5

6 Chapter 2. Problem Description

2.2 Restrictions

To reach its full potential, the PRO data would need to be synchronized with the existingelectronic medical records, so that clinicians can analyze it together with individual patientinformation [14, 15]. Furthermore, the system should allow patients to complete the ques-tionnaires periodically and view previous results, which would require log-in functionalitywith a database that can handle the users and store data. However, since transferring andstoring patient information is associated with high security which requires extensive work tointegrate, these functionalities was not implemented nor covered in this study due to timelimitations. Instead, main focus was placed on developing the basic components needed toevaluate the usability of the tablet-based questionnaire from a patient perspective.

Another time-based restriction concerns the number of questionnaires implemented inthe �nal prototype. While the ultimate goal is to have all six questionnaires (as used in theinitial study) available in the system, only questions from one of them, called the StrokeImpact Scale (SIS), was included. The SIS questionnaire consists of 12 di�erent parts,each containing a number of questions related to a certain domain, such as sensory ability,memory and thinking.

2.3 Requirements

Following are the requirements of the �nal prototype, prioritized in descending order ofimportance, according to the MoSCoW-method [16]:

Functional

� The system must contain questions, for the user to answer, from at least two di�erentparts of the SIS questionnaire.

� The system should be able to build, and for the user present, the Rehab-Compasschart based on the questionnaire answers.

� The system should be scalable, i.e. be easy to extend, e.g. by adding more question-naires and levels to the Rehab-Compass.

Non-functional

� The Graphical User Interface (GUI) must be adapted to a tablet format of size 9.7".

� The GUI should be responsive, i.e. the layout should adapt to di�erent screen sizes.

Chapter 3

Methods

The project was carried out in four phases, described below. See Figure 3.1 for an overview.

Thesis

1 Pre-study

1.1 Gather

background

information

1.2 Establish

design guide-

lines

1.3 Conduct

in-depth study

2 Design phase

2.1 Create

prototypes

2.2 Test

prototypes

2.3 Design

database struc-

ture

3 Implementation

3.1 Implement

database

3.2 Implement

questionnaire

framework

3.3 Implement

Rehab-

Compass chart

4 Evaluation

4.1 Prepare

user tests

4.2 Conduct

user tests

4.3 Evaluate

result

Figure 3.1: Work Breakdown Structure of the project.

3.1 Pre-study

During the project's initial phase, background information regarding ePRO systems andcharacteristics of the target group, i.e. stroke patients, was collected to establish a set ofgeneral design guidelines to be used as a starting point in the design phase. This was followedby an in-depth study exploring the potential and challenges of adaptive user interfaces, andhow these can be used to increase accessibility in heterogeneous user groups.

The pre-study was carried out through an online search in databases where peer-reviewedmaterial was prioritized, in order to ensure high quality and creditability.

7

8 Chapter 3. Methods

3.2 Design phase

With the pre-study as a basis, a database design as well as GUI prototypes of varying �delitywere produced in Axure RP and tested with users and domain experts. A user-centered anditerative design process was adopted to improve the quality and functionality of the design,meaning that step 2.1 and 2.2 of the Work Breakdown Structure in Figure 3.1 were cyclicallyrepeated. Section 3.2.1 describes the user testing conducted on the developed prototypes.

3.2.1 User testing

Two main user test iterations were conducted on low- and medium-�delity prototypes ofthe ePRO questionnaire. All user tests were performed individually with each user, andcompared two prototypes with di�erent designs, which are further presented as a part of thedevelopment process in Chapter 6. The prototypes were tested in random order to avoidbias, and all participants were informed about the purpose of the study, their right to declineor withdraw at any time, as well as their guaranteed anonymity.

Iteration 1: Low-�delity prototypes

The low-�delity prototypes were tested with four users (three men and one woman) aged59�62 years, and four domain experts, consisting of one doctor and a doctoral studentinvolved in the project and two occupational therapists. All users and domain expertscompleted the questions on both prototypes and verbally gave their opinions of which versionthey preferred and why. Observed problems during use were also noted.

Iteration 2: Medium-�delity prototypes

The medium-�delity prototypes were tested with a new user group, consisting of four se-niors (three male, one female) aged 65�78, having previously su�ered from a stroke. Theparticipants' task was to answer the same seven questions on each prototype and thereafteranswer the usability questionnaire described in Section 3.5.

After testing and rating both prototypes, the participants were orally asked which versionthey preferred and why � both in their current state and an imagined preference earlier intheir rehabilitation process. Encountered problems and whether the patients needed anytechnical assistance during task completion were also noted.

As a complement to the user tests, the prototypes were also orally reviewed by tworesearch nurses involved in several research projects related to stroke and with many yearsof experience with stroke patients.

3.3 Implementation

The �nal prototype of the system was implemented in Microsoft Visual Studio 2016 with C#,HTML5, CSS3, JavaScript libraries and the ASP.NET framework. A web-based solutionwas chosen since it is both platform- and device independent, and hence can reach out tomore users at a lesser cost, compared to native applications. Although access to Internetis required and overall performance is weaker [17], other reasons for choosing the alternatenative platform, such as access to hardware features like camera and sensors, were notrelevant in this type of application. Furthermore, with web-based platforms, there is no needto explicitly download and update the application, which could pose additional technical

3.4. Evaluation 9

challenges to the users [18]. However, as noted by Eremenco and colleagues [19], web-basedPRO instruments allow for a wide range of screen sizes and browsers, which increases therisk of response variations due to possible di�erences in how items are presented and therebyinterpreted.

More details regarding the implementation phase is given in Chapter 6.

3.4 Evaluation

This section describes and motivates the methods used to evaluate the �nal prototype ofthe ePRO questionnaire.

3.4.1 Study design

Coons and colleagues [18] describe three evaluation methods to consider when transitioninga paper-based PRO instrument into an electronic format: cognitive debrie�ng, usabilitytesting and equivalence testing. In cognitive debrie�ng, respondents are verbally interviewedto assess whether the ePRO adaptation changes how they interpret and respond to thequestions and instructions. Usability testing is used to examine whether respondents in thetarget group can use the software and device as intended. Lastly, equivalence testing isperformed to be assured that PRO scores from the electronic version are not signi�cantlydi�erent from those of the paper version.

The authors further describe how the extent of testing needed to ensure a successfulmigration depends on the degree of modi�cation done to the original PRO instrument. Themodi�cations made on the original paper-based PRO measure in this study were consideredto be minor, since no changes were made to item wording or response options. According tothe guidelines provided by the authors, minor modi�cations will not result in any signi�cantdi�erences in PRO scores, hence the evaluation in this study only needed to consist ofcognitive debrie�ng and usability testing. However, due to time limitations, only usabilitytesting was conducted, as it was considered the most relevant aspect in the scope of thisthesis.

3.4.2 Participants

The participants consisted of �ve patients, having su�ered from a stroke or TIA during thelast year, which is within the recommended range (5�10) when conducting usability test-ing of ePROs [18]. All patients had previously completed the paper-based versions of thequestionnaires, and were recruited in connection to a follow-up visit at the Neurorehabilita-tion clinic at Umeå University hospital. All patients were informed about the backgroundand purpose of the study, their right to decline or withdraw at any time, as well as theirguaranteed anonymity and gave their signed consent to participate.

3.4.3 Test setting, user task and procedure

The user tests were carried out in a separate quiet consulting room, together with theresponsible doctor or doctoral student. The patients' task consisted of completing two partsof the SIS questionnaire (previously described in Section 2.2), containing seven Likert-scalequestions and one Visual Analog Scale (VAS) question respectively. After completion, thepatients rated their satisfaction with the prototype on a usability questionnaire furtherdescribed in Section 3.5. Feedback and comments were also noted, as well as observed

10 Chapter 3. Methods

problems and whether the patients needed any technical assistance during use. Additionally,the patients' age, gender, time since stroke or TIA and experience with computers andtablets were collected.

3.5 Usability questionnaires

To assess the users subjective satisfaction with tested prototypes, a questionnaire coveringthe following usability aspects was used:

� Understanding of the system

� Touching the intended answer on the screen

� Reading the text on the screen

� Focusing on each question

These aspects were considered to be the most crucial usability factors in this speci�ctarget group, based on results from both related studies [20, 21], the conducted pre-studyand initial user testing. Each aspect was rated on a VAS ranging from 1 (very di�cult) to5 (very easy). Scores were rounded to the nearest half integer.

The usability questionnaire used in the �nal prototype also included four questions com-paring the tablet-based questionnaire to the paper-based questionnaire, by letting the pa-tients choose which one of them they felt was easier to understand, more comfortable to use,easier to read from and �nally their overall choice of preference. The option �No di�erence�was also included. These questions were adapted from a similar study conducted by Richteret. al [21], further described in Chapter 4, where tablet-based ePRO questionnaires forrheumatism patients was compared to paper-pencil versions. In their study, a �fth questionregarding the experienced completion-time was also included. While being an interestingaspect, that question was not applicable in this study, since the users did not complete thewhole questionnaire on the tablet.

3.6 Equipment

The tablet used during the development and user testing was an iPad Pro of size 9.7".

Chapter 4

Background

The use of ePRO measures as a way of empowering a patient-centered care in health care israpidly increasing as they become more accessible with the advances in technology and thewide spreading of Internet connectivity [22]. This chapter describes related work conductedin this �eld and ends with a summary of general design guidelines for constructing tablet-based ePRO questionnaires aimed at stroke patients.

4.1 Related work

While there seems to be lacking research regarding the usability of tablet-based ePROsystems aimed speci�cally at stroke patients, the area have been studied in other patientgroups.

In 2008, Richter et. al [21] evaluated the feasibility of using electronic data collectionof self-administered patient questionnaires for rheumatism patients, by comparing the re-sults to corresponding paper-pencil versions. A total of 153 patients with various types ofrheumatism completed a set of paper-based and electronic tablet questionnaires and ratedtheir experiences on six-graded Likert scales. The results showed that 62.1 percent of the pa-tients preferred using the tablet versions of the questionnaires in the future, while 4.6 percentfelt uncomfortable using the tablet due to their rheumatic disease. The authors concludethat the capturing of self-administered patient questionnaires via tablets can enhance thequality of care and communication between patients and doctors, as well as contribute topatients' self-empowerment in rheumatology.

A more extensive and general study, by Hess et. al [23], measured the usability of aself-administered tablet computer-based questionnaire. The questionnaire was implementedin a screening tool called Functional Assessment Screening Tablet (FAST) and used as apart of their routine patient care. Between January 2004 and January 2006, 10,999 patientsin a university-based primary care practice completed the questionnaire and reported theexperienced level of di�culty using the FAST. The results were analyzed based on a set ofdemographic factors, such as the patient's age, sex, race, education and medical conditions.A majority out of all patients (84 percent) reported having no di�culty using the FAST andonly three percent experienced a lot of di�culty. However, the results varied by age, raceand educational attainment. Patients 65 years or older reported having more di�cultiesthan the average patient, with 28 percent experiencing some di�culty and thirteen percenta lot of di�culty. The authors emphasize the importance of further research to identify thereasons why certain groups experience more di�culties, and whether those obstacles can be

11

12 Chapter 4. Background

easily overcome by repetition and guidance or if they require more e�ort.Kavalieratos and colleagues [20] evaluated a tablet-based ePRO system on 2,728 cardi-

ology patients, aged 63 years old on average, from August 2012 to August 2013. A minorityof the patients (18 percent) reported having di�culty using the system, with patients 65years or older and patients with rheumatoid disease being overrepresented, as well as thosereporting poorer health than a year prior. The most common reasons for reporting dif-�culty were tapping problems (47 percent) and item comprehension (14 percent). Otherissues identi�ed were visibility and physical limitations.

Apart from the usability aspect, the validity of electronic PRO measurements have alsobeen studied. Gwaltney, Shields and Shi�man [24] conducted a literature review to assesswhether computer-administered and paper-based PRO measures produce the same scores.The authors concluded there is extensive evidence indicating they are indeed equivalent,and furthermore listed the following reasons to why computerized assessments of PRO datamay be preferred over the traditional paper-and-pencil measures:

� Missing data can be reduced by requiring completion of each question.

� Computerized handling of complex skip patterns can ease use for patients and reduceincomplete or invalid data.

� Out of range and ambiguous data can be eliminated by only allowing selection of theon-screen options.

� The e�ort and human-errors associated with handling paper data can be reduced.

� Sophisticated design can be implemented to ensure a valid representation of the pa-tient's experience in diary studies.

� Compliance with diaries can be increased using computerized time-tags.

However, aside from the advantages listed above, the digitization also brings up newchallenges. Schick-Makaro� and Molzahn [25] summarized the following challenges relatedto electronic PROs, addressed in previous studies:

� Requires greater initial costs and planning.

� It may be necessary to train clinical sta�, researchers or patients to use the system.

� Potential resistance to electronic administration.

� Lack of technological resources, e.g. computers or mobile devices and secure servers.

� People with limited computer experience may need support.

� Linking the ePRO data to patient health records and integrating it into clinic work�ow.

4.2 Designing ePROs for stroke patients

While several studies [26,27] have shown how tablets, compared to other electronic devices,can be especially bene�cial when targeting elderly and disabled users, allowing for a naturalinteraction by simply touching directly on the screen, the graphical user interface (GUI) mustbe adapted according to their physical and cognitive impairments. This section describescommon characteristics of stroke patients and a summary of related design guidelines foundin the literature.

4.2. Designing ePROs for stroke patients 13

4.2.1 Stroke and its implications

A stroke is a brain injury, also referred to as a brain attack, most commonly caused bya sudden blockage within an artery that supplies blood to the brain [2]. About half of itssurvivors is left disabled, with various degrees of physical, emotional and cognitive problems,depending on what part of the brain was damaged and how severely [28]. Some of the mostcommon conditions that may develop post-stroke are listed below, based on informationfrom the Stroke Association [29].

� Physical : visual problems, lack of energy/fatigue, muscle weakness and coordinationproblems, pain, incontinence.

� Emotional : feelings of anger, frustration, sadness, anxiety and other depression-likesymptoms.

� Cognitive: memory problems, language di�culties, trouble reading and writing.

Many of these conditions correlate with age-related impairments, such as visual defectsand memory limitations, which are also important to consider, since a majority (75 percent)of stroke survivors are over 65 years old [30]. Becker and Webbe [31] developed a usabilityframework for older users of handheld technology and listed the following aging factors toconsider in the design:

� Vision: reduced ability to focus on close objects, decline in visual acuity, i.e. theclarity of vision, decreased light sensitivity, increased sensitivity to glare and reduceddepth perception.

� Hearing : loss of sensitivity to high-frequency sounds and reduced ability to discrimi-nate similar sounds and localize objects with low frequency sounds.

� Cognition: Reduced ability in problem solving, working memory, attention and con-cept formation.

� Motor skills: Decreased motor coordination.

� Literacy : Decline in language comprehension.

� Technology skills: While the computer use has increased in this user group, manyseniors still have limited experience with newer technology and may not use it regularly.

4.2.2 Design guidelines

While stroke patients in no way should be seen as a homogeneous user group, they do sharemany characteristics that have certain implications on the design. Table 4.1 presents a sum-mary of existing design guidelines found in the literature, including both graphical aspectsand other factors relevant when targeting people with physical and cognitive impairments,as well as when designing ePRO questionnaires in general.

14 Chapter 4. Background

Table 4.1: Design guidelines for ePRO questionnaires aimed at stroke patients.

Aspect Guidelines

Graphical� Text : Use simple sans serif fonts, e.g. Helvetica and Arial, of size 12-14 points, left-justify the text and highlight important information[32].

� Color : Use warm colors and avoid combinations of blue and yellowor red and green tones [33]. Avoid rapid changes in brightness andlow contrast between foreground and background [32].

� Layout : Concentrate information mainly in the center of the screenand be simple, clear and consistent [32].

� Interactive elements: Use big and clear targets [32, 34]. Targetsshould be 8 mm or larger in size to avoid tapping errors [35]. Radiobuttons should explicitly indicate that the clickable area containsboth the button and its text [36].

� Graphics: Only use graphics when necessary and avoid animations[32]. Provide meaningful icons together with labels [33].

� Navigation: Provide clear navigation cues [32].

Structural � Question and response options should be available on the samescreen [36].

� Avoid multiple windows [32].

� Show a progress indicator of how far the patient is through thequestionnaire [36].

� Avoid scroll bars [32,36,37].

Functional � Allow patients to skip individual questions [18].

� Implement the ability to go back and change prior responses [18].

� Prevent users from unintentionally missing questions to ensure datacompleteness [19].

Other � Reduce working memory demand by limiting the number of choicesand supporting recognition rather than recall [32].

� Textual information should be presented in the patient's language[36].

Chapter 5

Designing for diversity

As the Internet and digital technology are increasingly integrated as a part of our everydaylife, dealing with diverse users, devices and environments has become a central design chal-lenge � especially when targeting heterogeneous user groups, such as elderly and disabled,whose needs and abilities can be very di�erent and change over time [38].

There are basically two contrasting approaches to this problem. One is trying to �nd asingle user interface design that works well for everyone, related to the concept of UniversalDesign, de�ned as �the design of products and environments to be usable to the greatest extentpossible by people of all ages and abilities� without the need for adaptation of specializeddesign [39]. The other is creating an intelligent system that can tailor to individual users andthus provide various user interface designs for the same functionality. The later approachhave recently been promoted as the better solution to the problem of context variability,with arguments such as enhanced usability [40] and increased accessibility [41].

This chapter presents the concept of adaptive user interfaces, discusses its potential andchallenges, and provides real examples of how it can be used to increase accessibility inheterogeneous user groups.

5.1 Adaptive user interfaces

Adaptive user interfaces are not to be confused with adaptable user interfaces, which refer tointerfaces that can be customized by users to suit their speci�c needs [42]. For example, inMicrosoft Word 2016, users can choose what buttons or commands to be shown in toolbarsand menus. Adaptive user interfaces, on the other hand, are de�ned by their ability to:

...model the individual user's interests, preferences and usage characteristics toallow the interface to tailor itself to each user [42].

The goal is to have the interface adapt to the user, instead of the other way around [43].While being a subject of research ever since the 80's [41], these types of systems havebecome particularly relevant today because of their ability to automatically tailor to theuser's preferences and surrounding conditions at runtime [44]. This process of system-drivencontent tailoring is often referred to as personalization, which should be distinguished fromthe user-driven customization related to adaptable interfaces [45].

15

16 Chapter 5. Designing for diversity

5.1.1 Achieving personalization

Personalizing interfaces is a challenging and complex task. Every user is unique, with theirown requirements and characteristics, and even though accommodating to each and everyindividual might not be fully realistic, there are ways of approaching the problem to createa similar behavior.

Germanakos and colleagues [46] describe the method of user segmentation, i.e. subdivi-sion of the user population �into more or less homogeneous, mutually exclusive subsets ofusers who share a common user pro�le characteristics� such as:

� Demographic characteristics (e.g. age, gender, urban or rural based, region)

� Socio-economic characteristics (e.g. income, class, sector)

� Psychographic characteristics (e.g. life style, values, sensitivity to new trends)

� Individual physical and psychological characteristics (e.g. disabilities, attitude)

The authors further describe how the user pro�le can contain either static data, thatrarely changes (such as demographic information), or dynamic data, that frequently changes.This data can be retrieved either explicitly, using online forms and questionnaires andthereby result in static data, or implicitly, by continuously registering the user's naviga-tional behavior. For example, the user interface could adapt according to the user's speedand accuracy in mouse clicks and hovering [47].

In addition to the user pro�le, the authors point out the need of adapting the systemto the characteristics of the used device, by also providing a device pro�le. Current bat-tery level, network connectivity, screen orientation and screen brightness are examples ofproperties that could be included [48].

Contextual factors

Another important factor to consider in today's mobile and portable applications is thecontext of use, that is the conditions under which a product is used [49]. Castillejo, Almeidaand López-de-Ipiña [50] mention natural lighting, ambient noise, current temperature anduser movement as contextual aspects that can a�ect the user's capabilities. The authorssimilarly point out how context can in�uence functionality in the used devices, such asnetwork bandwidth and location accuracy.

5.1.2 Techniques: Adaptive presentation and navigation

Ramachandran [51] describes two techniques that have been commonly used to create adap-tive interfaces: adaptive presentation and adaptive navigation. While both techniques basethe adaptation on user characteristics, such as current level of knowledge and competence,they have di�erent goals. Adaptive presentation is about adapting the content of the page,by only providing information relevant and understandable to the user. For example, low-level details can be hidden for novice users, while additional information can be excludedfor expert users. The goal of adaptive navigation on the other hand, is to guide users inaccomplishing certain goals in the application.

The author further describes how these techniques can be implemented by setting up arelational database that stores presentation rules and user pro�les, and having a processinglogic unit responsible for selecting the appropriate presentation or navigation strategy basedon the current page and user pro�le.

5.2. Examples: Using adaptivity to increase accessibility 17

5.1.3 Potential and challenges

In theory, a user interface that automatically adapts to the user's individual needs andabilities sounds very appealing, with potential user interaction improvements like easingsystem use, reducing cognitive load and facilitating user performance [52]. However, creatingsuch systems requires a lot of e�ort and expertise to be successful in practice.

As noted by Findlater and Gajos [53], the bene�ts gained from accurate adaptations mustbe weighed against the costs of inaccurate adaptations. For example, one could imaginethe frustration of a sudden change in the user interface that does not even make sense.Although a possible adaptation detected by the system does not necessarily need to beapplied abruptly, undesirable adaptations that may occur is still a major problem.

A �exible and dynamically changing user interface also comes with certain usability sidee�ects, regardless of the accuracy of the adaptation. The inconsistency can be confusingto users and result in a feeling of lost control, decreasing both acceptance and usability[54]. Similarly, the unpredictability poses challenges to developers and designers, since themany interaction variations complicates system evaluation [53]. Additionally, collecting andstoring data about individual users can also raise privacy concerns [53,55].

The value of adding adaptivity to a system also depends on other variables. Lavie andMeyer [52] suggest the task, the user, the situation and the level of adaptivity are keyfactors to consider. For example, the authors discuss how complex or frequently performedtasks may bene�t more from adaptive user interfaces than easy or non-routine tasks. Theyillustrate this by an adaptive menu, where frequently used items can be displayed in thebeginning of the menu to be more accessible, while the same support can not provided whenthe user wants to search for rarely or never used items.

Furthermore, the authors point out that the addition of adaptivity should not be anall-or-nothing decision, i.e. either a fully adaptive or entirely manual system. Instead, theyargue how di�erent levels of adaptivity should be considered to involve the users in theinteraction, and thereby increase both controllability and predictability. These types ofdesign approaches are referred to as mixed-initiative strategies, which combine adaptive andadaptable approaches to get the best out of two worlds � the high user-control of adaptableinterfaces and the time- and e�ort saving advantages of adaptive interfaces [56].

5.2 Examples: Using adaptivity to increase accessibility

This section presents two concrete examples where adaptivity have been added to systemsas a means to increase accessibility in heterogeneous user groups.

5.2.1 The SUPPLE system

The SUPPLE system, able to automatically at run-time generate a wide variety of possibleinterfaces adapted to the user's device, tasks, preferences and abilities, was presented byGajos, Weld and Wobbrock [41].

The authors describe a user evaluation of the system, conducted with 11 motor-impairedparticipants and six able-bodied participants on a laptop, using each participants' input de-vice of choice. Before the main experiment, a predictive model of the participants' motorabilities was built, based on their observed performance in four basic tasks including point-ing, dragging, list selection and multiple clicking. For the main experiment, three di�erentapplications were used: a font-formatting dialog box, a print dialog box (both resemblingthat of Microsoft Word 2003) and a synthetic version containing data types commonly seen

18 Chapter 5. Designing for diversity

in dialog boxes. Additionally, each application had three di�erent interface variants: onedefault variant, re-implemented to resemble the original or �typical� design, and two au-tomatically generated variants based on the participant's previously stated preference andmeasured motor ability. The participants performed a number of di�erent tasks, with oper-ations like clicking a button or setting a widget value, on each application and correspondinginterface variant.

Results showed that the participants were both faster and less error-prone when usingthe automatically generated interfaces, compared to the default ones, with a reduced e�-ciency gap of 62 percent between motor-impaired and able-bodied participants [41]. Theautomatically generated interfaces were also strongly preferred by the participants withmotor-impairments. The authors clearly point out that the SUPPLE system is not sup-posed to replace human designers, but work as a means to provide �non-typical� users withalternative user interfaces and thereby e�ectively increase both accessibility and scalability.

5.2.2 MyUI

Peissner and colleagues [54] created a system called MyUI, that generates adaptive userinterfaces at run-time to support accessibility. Based on the characteristics of the currentuser, device and surrounding environment, the system holds a user interface pro�le thatde�nes the general appearance of the user interface and controls variables such as font size,amount of displayed text versus graphics and activation of voice input. The user interface isadapted according to prede�ned design patterns, for example by providing visually impairedusers with larger font sizes and displaying iconographic information to users with di�cultiesin language reception. To increase controllability and transparency, the system lets theuser explicitly decide whether to apply a proposed adaptation or not, and animates thetransitions between the former and new interface. While the adaptations in MyUI also canbe triggered manually by the user, making it a mixed-initiative system, the authors arguethat the automatic adaptations should be the most bene�cial when it comes to users withdisabilities and lacking ICT experience. However, their initial user study indicated that theconcept of adaptation could to be di�cult to understand and recognize in that particulartarget group [54].

Chapter 6

Development process

This chapter describes the design and implementation phase of the ePRO system, the deci-sions made and the iterations of prototypes and user tests leading up to the �nal version.Finally, a summary of how well requirements were met is presented.

6.1 Choosing the questionnaire format

The transition from a paper-based PRO instrument to an electronic version usually involvessome degree of modi�cation of instructions and the way items are presented to �t the newformat. As pointed out by Eremenco and colleagues [19], the electronic version does notnecessarily need to look exactly the same as the paper version, although item and responsetext should be retained in order to produce comparable result.

6.1.1 Item presentation

There are basically two ways of presenting items in an electronic questionnaire: one itemper screen or multiple items per screen. The paper version of the questionnaire in this studyuses a grid-based format, where multiple questions are grouped together based on theirdomain, as seen in Figure 6.1 below.

Figure 6.1: Question group as presented in the original paper questionnaire.

19

20 Chapter 6. Development process

Since the goal is to primarily adapt the ePRO system for use on a tablet device, themultiple question grid-based layout as in the original paper version is possible to retain,whereas the more limited screen size of a smartphone would likely require the one-item-per-screen approach. However, as noted by Eremenco and colleagues [19], if the PRO instrumentis supposed to be used across various screen sizes, which is inevitable with a web-basedplatform, consistency in the item presentation is recommended to avoid potential responsedi�erences caused by the format. For this reason, the one-item-per-screen approach wouldbe the better choice.

Additionally, asking one question per screen has been suggested to reduce cognitiveload [57], which could be especially important in this target group. Multiple questions perscreen have also been shown to increase the risk of item non-response, especially in olderrespondents [58]. On the other hand, separating every question results in more screens,which requires more physical actions, in this case tapping, from the user [59]. This mayresult in longer completion times [58] and could potentially increase tapping problems, asmotor and technology skills might be reduced for a majority in this target group.

6.1.2 Response options

In the paper-based version, patients mark their responses on �ve-graded Likert scales, aspreviously shown in Figure 6.1, in all questions except from the last, where patients ratetheir overall perceived rehabilitation status from 0-100 on an anchored Visual Analog Scale(VAS), see Figure 6.2.

Figure 6.2: The VAS used in the last question of the original paper questionnaire.

While there are many other ways of formatting response options, it was decided tomaintain the original response types to keep the text modi�cations as small as possible, assuggested by Eremenco et. al [19].

6.2. Iteration 1: Low-�delity prototypes 21

6.2 Iteration 1: Low-�delity prototypes

To be able to test how users handle and experience the di�erent presentation formats, twolow-�delity prototypes were produced. Regardless of the presentation format, it was decidedthat neither of the prototypes should require any scrolling, all text should be left-justi�edand of sans-serif type and a progress bar be included on both (based on the previouslypresented design guidelines). Furthermore, the prototypes were both adapted to a horizontalorientation of the tablet, to best exploit the available screen size and avoid the need ofscrolling. Additional functionality and graphical design choices, such as color and form,were not speci�cally considered in this iteration. Focus was mainly to create the componentsneeded to test the di�erent ways of presenting items.

The number of questions included in each prototype was limited and only covered ques-tions from the �rst two domains in Likert scale-format and the last question with the VAS.The VAS had the same appearance on both prototypes. As seen in Figure 6.3, it wasconverted to a horizontal slider. Both prototypes also included an introduction to the ques-tionnaire.

Figure 6.3: The VAS used in both prototypes.

6.2.1 Prototype 1 � One item per screen

Questions in the one-item-per-screen approach were presented as seen in Figure 6.4.

Figure 6.4: Prototype presenting one item per screen.

To keep the logical grouping by domain, each question group got to represent a certainpart, with the current part and related description displayed on top of each screen. To


further guide the user through the questionnaire and its parts, an overview was presentedat the beginning of the questionnaire and after completion of each part, see Figure 6.5.

Figure 6.5: Overview presented in the one-item-per-screen-questionnaire.

6.2.2 Prototype 2 � Multiple items per screen

As seen in Figure 6.6, the multiple-items-per-screen prototype presented the questions in agrid-based layout, similar to the paper-based version of the questionnaire, where questionsare grouped together by domain.

Figure 6.6: Prototype presenting multiple items per screen.

6.3. Iteration 2: Medium-�delity prototypes 23

6.2.3 User tests

As described in Section 3.2.1, the prototypes were tested with four users (three men and onewoman) aged 59�62 years, and four domain experts, consisting of one doctor and a doctoralstudent involved in the project and two occupational therapists.

Results

A majority of the users preferred the multiple-items-per-screen approach, with the mainarguments being it gave a better overview, was faster to complete and had a nicer design.However, all domain experts, except from one of the occupational therapists, thought theone-item-per-screen approach could be preferable in patients experiencing cognitive andmotor de�ciencies, making it easier to focus on each question and touch the intended answerarea. The occupational therapist not liking this approach felt the layout was too messy.

Another observed problem with the one-item-per-screen prototype was that some userstouched the `Next' button multiple times, not noticing they had already progressed to thenext screen, which resulted in unintended missed questions. This could be due to the visualand textual information between subsequent question screens being so similar. The lack ofcolor and other graphical design elements may also have a�ected the user experience.

Additionally, a few occasions of tapping problems were noticed, due to the elastic scrollingof the interface, also called the rubber band e�ect, where the screen �bounces� back whenreaching the top or bottom of a page. When trying to mark their answers, or moving on tothe next question, users sometimes touch-dragged on the screen, triggering this bounce e�ectinstead of the intended action of selecting a button. This happened on both prototypes, butmainly on the one-item-per-screen version, since it required twice as much touch interactions(�rst selecting an answer and then touching the `Next' button).

Regarding the VAS, no direct usability problems were observed. However, both usersand domain experts thought that the exact position of the marker should be more obviousand possible to change also by touching directly on the targeted position, since draggingcan be a problematic interaction for users with motor de�ciencies.

6.3 Iteration 2: Medium-�delity prototypes

Taking into account the feedback gained from the �rst user test, the design of the one-item-per-screen prototype was re-built, to make it more appealing and easier to use, see Figure6.7.

(a) State when question is unanswered. (b) State when question is answered.

Figure 6.7: Two di�erent states of a question screen in the second iteration of the one-item-per-screen prototype.


While still keeping all text left-justi�ed, the main textual information was centered onthe screen, satisfying also the layout design guideline. Shapes of buttons and graphicalelements were rounded to create a smoother appearance. Rounded corners have also beenstated by experts to take less cognitive e�ort to process [60, 61], which is an even strongermotive to use them in this target group. The progress bar was also changed to rely more ona graphical rather than textual representation, assisting the discrimination of subsequentquestions. As seen in Figure 6.7a, the progress bar shows the current question in orangewhen unanswered, and in green when answered (Figure 6.7b), to indicate completion. Thecolor-shift feedback was also used for the `Next' button, which stays gray until the questionis answered and shifted to green.

To further avoid unintended missed items, the `Next' button was disabled until thecurrent question was answered. While this ensures full completion of the questionnaire, italso violates the design guideline of allowing patients to skip questions. After discussionsduring project meetings, this was still decided to be the best solution, at least temporarily,since a `skip' option could potentially be overused and alternative ways would be too time-consuming to implement during this project.

Colors were also added in the multiple-items-per-screen approach, see Figure 6.8 below,as well as the new, cleaner and simpler header with an included, yet not functioning, optionto end the questionnaire. Additionally, vertical scrolling was disabled on both prototypesto prevent the rubber band e�ect, causing tapping problems in the previous user tests.

Figure 6.8: Second version of the multiple-items-per-screen prototype.

The VAS was not considered in this iteration, since the improvements were not possibleto create in Axure RP.

6.3.1 User tests

As previously described in Section 3.2.1, these prototypes were tested with a new user group,consisting of four seniors (three male, one female) aged 65�78, having previously su�eredfrom a stroke.

6.3. Iteration 2: Medium-�delity prototypes 25

Results

As seen in Table 6.1, in their current state, two users preferred the multiple-items-per-screenprototype, one user the one-item-per-screen prototype, and one user stated either one was�ne.

Table 6.1: Users and their choice of prototype.

User Gender Age Years sincestroke

Computer andtablet

experience

Currentprototypepreference

Earlierprototypepreference

User 1 Male 66 6 None; AverageMultiple-items

One-item

User 2 Male 66 4 None; None One-item One-item

User 3 Female 74 16 Average; ExpertMultiple-items

One-item

User 4 Male 65 6 Average; Average Either one One-item

Although not disliking the one-item-per screen approach, User 1 and User 3 appreciatedthe overview and sense of control achieved when being able to see multiple questions perscreen. However, when asked what item presentation approach they thought they wouldhave preferred earlier in their rehabilitation process, all participants assumed one-item-perscreen would have been easier to handle and process due to the post-stroke symptoms ofmental fatigue and concentration di�culties. User 2, the participant most recently havingsu�ered from a stroke and with no experience of either computers or tablets, clearly statedthe one-item-per screen prototype was a lot easier to use and understand. He felt like themultiple-items-per-screen prototype was �too messy�, making it harder to distinguish andconcentrate on each question.

The participants' comments were consistent with the subjective quantitative results, seenin Table 6.2 below. All users rated the usability aspects on both prototypes between 4 and5, except User 3, who consistently rated the multiple-items-per-screen prototype between 2and 3, decreasing its average scores.

Table 6.2: Average user rating (1=very di�cult; 5=very easy).

Usability aspect One item Multiple items

Understanding the system 4.375 4

Touching the intended answer on the screen 4.375 4.125

Reading the text on the screen 4.625 3.875

Focusing on each question 4.375 3.9

After being instructed once on how to use the systems, all participants except one couldcomplete the seven questions without any technical assistance. User 2 once marked anunintended answer on the multiple-items-per-screen prototype and did not know how tochange it. User 3 noted that it would be good to have available instructions on those kind


of technical dilemmas and also thought the overview, listing all parts of the questionnaire,could potentially be overwhelming for some users. Although not related to the usability ofthe system, User 4, who liked both prototypes, felt like some questions were too general anddepended on the situation, e.g. �How di�cult is it for you to talk on the phone?�, makingit hard to know what to answer.

Figure 6.9 shows a user testing the multiple-items-per-screen prototype.

Figure 6.9: A user testing the multiple-items-per-screen prototype.

The two domain experts were positive to the use of a tablet for collecting PRO data.After having tested the two prototypes, they were both convinced that the one-item-per-screen version would be the best choice for this target group, stating multiple questions perscreen would be far too confusing and hard to use due to the overload of information anddemand of higher touch precision. The only thing they were a bit concerned about was theoverview listing all parts of the questionnaire, seen in Figure 6.5. They thought it could betoo much to handle and maybe not necessary, alternatively be presented in another way.

6.3.2 Conclusion

Although the user sample is too small to make fully reliable conclusions, it was decided toimplement the one-item-per-screen approach, since that version, when improved during thesecond iteration, worked well for all participants � regardless of age, post-stroke conditionand experience with computers and tablets. Arguments from both the domain experts andprevious studies also indicate that presenting one item per screen is better, both whenit comes to this particular target group, as well as electronic questionnaires in general.The advantages of a potentially faster completion time and sense of control given by themultiple-items-per-screen approach seem to get the opposite e�ect in patients earlier in theirrehabilitation process, which is the considered target group in this study.

6.4. Implementation 27

6.4 Implementation

The system to implement consisted of three main components: a database to store ques-tionnaire data, a framework to present the questionnaires and the Rehab-Compass radarchart to visualize the questionnaire results.

6.4.1 The database

A SQL database was constructed based on the design structure seen in Figure 6.10 below.

Figure 6.10: Database schema.

Following is a short description of the database components, grouped by color:

� Users: A user can be either a patient or a clinician. Relevant user information isstored, such as personal identi�cation number and name of the patient.

� Sessions and answers: A new session is created when a user starts a questionnaireat a speci�c registered time. The answers given by the user are linked to the currentsession.

� Questionnaires and questions: A questionnaire consists of a number of parts,each corresponding to a certain category in the Rehab-Compass chart. Furthermore,every part contains a number of questions, each representing a certain section in theRehab-Compass chart. A question can also have an introductory text, speci�ed as


the question header, which can be reused for many questions as it is a separate table.The possible answer options to a question and their weighted value (stored in JSONformat, for example as {�5: Not at all di�cult�, �4: Somewhat di�cult�...}), as well asthe question type, e.g. Likert scale or VAS, are also speci�ed in their own tables andcan be related to many questions.

� Compass categories and sections: The Rehab-Compass chart consists of a numberof categories, which each contains a number of di�erent sections. For example, Com-munication, and Memory and Thinking, are both sections related to the Cognitioncategory.

Since log-in functionality was excluded, as previously described in Section 2.2, neitherthe User, Session nor Answer table was implemented, hence the �nal prototype did notinclude any permanent storing of the answers given by the current user.

6.4.2 The questionnaire framework

The one-item-per-screen layout built in Axure RP was recreated as a web-based version,with some minor changes. The overview, shown in-between two parts of the questionnaire,was highly simpli�ed to reduce the risk of cognitive overload, as suggested during the earlieruser tests. Instead of listing all included parts, it was changed to only inform the user whichpart had been completed, e.g. �You have now completed part 1 of 12.�, and how to moveon to the next part. To further minimize cognitive load, the labels below the navigationbuttons were removed, as the graphical representation with arrows were seen as su�cientnavigation cues.

The main priority was to implement the most necessary functions to be able to evaluatethe usability of the questionnaire and create a good user experience. The following sectionsbrie�y describe the most important implementation parts.

Dynamic creation of elements and responsiveness

To create a dynamic and scalable questionnaire framework, all textual information relatedto the questionnaire was accessed from the database. The progress bar was also built tobe dynamic, by adapting it to the number of questions in the current part. Additionally,Bootstrap was used to make the website responsive and adaptable to di�erent screen sizes.However, the layout was only optimized for a tablet and, while decent looking on computerscreens, it does not adapt well to smartphone screen sizes.

AJAX loaded content

When going back and forward between questions in a part, AJAX was used to communicatewith the server without the need of reloading the whole page � thereby o�ering a faster andbetter user experience. However, this requires the browser to support and have JavaScriptenabled to work. The same goes for other interactive elements in the interface, such as theprogress bar, built using Bootstrap's slider component, which is updated through jQuery.Another consequence of having AJAX loaded content is that the `Previous' button cannot be used as a traditional back button, since that will reload the page and discard allJavaScript updates. To still allow the user to return to the previous question state, thisfunctionality had to be manually implemented. Initially, attempts were made to push dataonto the browser history as the user proceeded through the questionnaire. However, dueto di�erent handling of browser history between di�erent browsers, this solution was not

6.5. Ful�llment of requirements 29

successful. Instead, the current question was stored in a session variable and retrieved andupdated from the server through AJAX calls, while the progress bar and other changeableelements were updated through jQuery when going back and forth in the questionnaire.

Reducing tapping problems

Since tapping problems had been speci�cally reported when designing for elderly user groupsin previous studies, and also observed during the early user tests of this study, much e�ortwas put into preventing such issues. Following is a summary of both graphical and technicalsolutions made to reduce tapping problems in the web-based version:

� Making target elements large enough

� Preventing the rubber band e�ect

� Disabling zoom through double tap

� Removing the 300 ms tap delay, apparent in some mobile browsers

� Disabling the `Next' button until the current question has been answered

� Using distinct color shifts to separate di�erent states

6.4.3 The Rehab-Compass chart

A web-based version of the Rehab-Compass chart was implemented using the open-sourceJavaScript chart library Chart.js. Since a basic design for how to present the multidimen-sional data from the questionnaire already had been created in Excel (earlier shown in Figure1.1), no additional prototypes and alternatives were developed before implementation. How-ever, some changes were made to improve design and comprehension of the chart. Apartfrom making the labels understandable to patients, i.e. translating from English to Swedishand replacing medical terms, the diagram lines were expanded into �elds, to provide a bet-ter overview. All modi�cations were discussed during project meetings with RehabilitationMedicine and MT-FoU.

Again, due to the user answers not being permanently stored, the chart was not linkedto any questionnaire data in this version.

6.5 Ful�llment of requirements

Based on the presented development process, Table 6.3 summarizes how well the speci�edrequirements of the �nal prototype were ful�lled.


Table 6.3: Requirements of the �nal prototype and whether they were ful�lled.

Requirement Priority Yes Partially No

The system contains questions, forthe user to answer, from at least twodi�erent parts of the SIS question-naire.

Must X

The system is able to build, andfor the user present, the Rehab-Compass chart based on the ques-tionnaire answers.

Should X

The system is scalable, i.e. easy toextend, e.g. by adding more ques-tionnaires and levels to the Rehab-Compass.

Should X

The Graphical User Interface (GUI)is adapted to a tablet format of size9.7".

Must X

The GUI is responsive, i.e. the lay-out adapts to di�erent screen sizes.

Should X

Chapter 7

Results

This chapter presents the �nal prototype and the results from the user evaluation.

7.1 Final prototype

The introduction, overview and several question screens from the �nal prototype, as pre-sented on an iPad Pro tablet device, are shown in Figures 7.1�7.5.

Figure 7.1: The introduction screen.

31

32 Chapter 7. Results

Figure 7.2: A completed Likert-scale question.

Figure 7.3: A not completed Likert-scale question.

7.1. Final prototype 33

Figure 7.4: The overview presented in-between two parts.

Figure 7.5: A not completed VAS-scale question.


Figure 7.6 below shows the �nal web-based version of the Rehab-Compass radar chart, as anexample and imagined patient's experienced level of stroke recovery on a scale from 0�100in di�erent health aspects. Each color �eld represents a certain category. For example, datawithin the pink �eld are related to mobility functionality, while data within the green �eldare related to di�erent aspects of mood. The placement of the labels showing these colorcodings was not �nalized.

Figure 7.6: The web-based version of the Rehab-Compass chart.

7.2 User evaluation

The �nal prototype of the questionnaire was tested with �ve patients, having su�ered froma stroke or TIA during the past year, see Table 7.1 below. None of them had participatedin the earlier user tests of this study.

Table 7.1: Users participating in the �nal user test.

User Gender AgeMonthssince

stroke/TIA

Computerexperience

Tabletexperience

User 1 Male 67 5 (stroke) Average Average

User 2 Female 53 9 (TIA) Average Average

User 3 Male 85 8 (TIA) Average Beginner

User 4 Male 81 5 (stroke) Beginner Beginner

User 5 Male 78 8 (stroke) Average Average

7.3. Feedback, comments and observations 35

Table 7.2 below shows the user ratings of the �nal prototype.

Table 7.2: Average user rating (1=very di�cult; 5=very easy).

Usability aspect Average rating (min-max)

Understanding the system 4.2 (3.5�5)

Touching the intended answer on the screen 4.2 (3.5�5)

Reading the text on the screen 4.2 (3.5�4.5)

Focusing on each question 3.9 (3�4.5)

Figure 7.7 below shows the distribution of users based on their preferences between thetablet- and paper-based questionnaire.

0 1 2 3 4 5

do you prefer?

is easier to read from?

is more comfortable to use?

is easier to understand?

Number of users

Which questionnaire...Tablet version No di�erence Paper version

Figure 7.7: Comparison between the tablet- and paper-based version of the questionnaire.

7.3 Feedback, comments and observations

All users understood how to answer the questions and how to continue to the next. However,reoccurring occasions of tapping problems, where the click/tap event was not triggereddue to incorrect timing/pressure, were observed with less computer- and tablet-experiencedusers.

Following are some of the user comments (translated from Swedish to English) notedduring the user tests:

� �This is a lot easier than dealing with all the papers!� (User 1)

� �This was fun, don't you have any more questions?� (User 4)

� �I like the way questions are presented, it gives me a good �ow.� (User 2, referring tohaving the variable part of the question separated from the �xed part by displaying iton a separate line and with a regular type face.)

User 2, who had su�ered a lot from mental fatigue, also expressed how the paper-basedquestionnaires were overwhelming and that the tablet version presenting one question at a


time, if possible to save and �pause�, would be a lot easier to handle. However, the user feltthe introduction was too text-heavy and suggested dividing it into di�erent screens or makingit shorter to increase readability. Similarly, the user thought that the VAS question and itsinstructions should be simpli�ed and recommended to remove all unnecessary informationon each question screen, e.g. stating the current questionnaire and part name. User 5,who as well had been strongly a�ected by post-stroke fatigue, also commented that theintroduction was too long, making it di�cult to read. User 3, though not reporting anysigni�cant problems with mental fatigue, noted that the instruction was just on the vergeof being too long.

Following is a summary of other topics and suggestions discussed during the user tests:

� Consider using numbers instead of letters on the progress bar.

� Provide separate instructions on paper instead of integrating more information in thesystem.

� The user should automatically return to where he/she ended when continuing thequestionnaire at a later time.

� The meaning of the `Quit questionnaire' button can be confusing � better to call it`Save and quit questionnaire'.

� Text-to-speech, sound feedback and pictures could be good complementary functions.

Chapter 8

Discussion

The transition from paper-based PRO questionnaires to electronic tablet-based versions haspreviously been shown to be successful in other patient groups [20, 21, 23], and the resultspresented in this thesis also give positive indications of its potential and usability whentargeting stroke patients. Although the user samples are too small to draw any reliableconclusions, they still provide some interesting insights to be discussed in this chapter.

8.1 Potential and usability of the ePRO questionnaire

The �ve stroke and TIA patients participating in the �nal user evaluation gave generallyhigh usability ratings (M=3.9�4.2, 1=very di�cult; 5=very easy) regarding the understand-ing, readability, touch interaction and ability to focus when using the tablet questionnaire.Additionally, a majority of the patients (four out of �ve) preferred the tablet-based ques-tionnaire over the paper-based version, stating it was more comfortable to use and easier tohandle, which is consistent with previous studies in other patient groups [20,21,23]. Resultsfrom initial user testing and interviews with research nurses in this �eld also indicated thatthe tablet-based ePRO questionnaire could be both applicable and usable in this targetgroup.

The challenge of using ePROs in elderly user groups, as they generally have lower ICTexperience, has been pointed out in previous studies [20, 23, 31]. Some of the seniors par-ticipating in the user tests of this study, both during the development process and �nalevaluation, indeed had little to no experience of using computers and tablets, some evenexpressing a resistance to modern technologies. One of them also preferred the paper-basedquestionnaire over the tablet-based version. However, all of them were still able to completethe questions, with or without some degree of technical assistance, and commented it wasfun and easier than they thought. This tells something about the intuitiveness of tabletsand their potential in elderly user groups, which have also been emphasized in other stud-ies [26,27]. Coming generations of elderly is also likely to be more familiar with tablets andother ICT devices, as digital technology becomes more widespread and integrated in societyand everyday life.

Yet, there is de�nitely room for improvement of the �nal prototype to further enhanceuse. Although reducing tapping problems was speci�cally prioritized during the developmentprocess, as it was reported as a common issue in a related study [20], patients with lesscomputer and tablet experience still had some trouble applying the right amount of pressureto trigger a button click. Whether this is something that can be overcome with practice or

37

38 Chapter 8. Discussion

not, the buttons should be optimized to be more touch sensitive, since tapping problemsalso may be due to age or stroke related impairments that might not disappear over time.

Displaying the appropriate amount of text on the screen was another challenge emergingalready during the �rst user tests and still present in the �nal evaluation. Too much informa-tion can be overwhelming, while too little information can lead to uncertainty. This dilemmawas discussed initially in the development process, when choosing the item presentation ap-proach. In retrospect, continuing with the one-item-per-screen version was de�nitely theright decision, primarily because it, as suggested in a previous study [57], reduces cognitiveload � a seemingly crucial usability factor in this target group with high prevalence of men-tal fatigue. While the ability to focus on the questions was rated relatively high (M=3.9),several patients still expressed that some parts were too text-heavy, which complicated read-ing. Although the textual content of the questionnaire was deliberatively kept unchanged(except from minor changes to instructions), these parts, e.g. the introduction and VASquestion, should be rewritten and simpli�ed � short but informative seems to be the wayto go. Text-to-speech functionality, sound feedback and pictures were also brought up aspotentially e�ective complementary output channels to further reduce cognitive load.

8.2 Increasing accessibility of the system

While the users participating in this study were generally positive to the tablet-based PROquestionnaire, a larger user sample is needed to increase reliability and cover more of thediversity seen in this target group. Only the few stroke patients that participated in thisstudy showed a wide range of post-stroke symptoms and experiences, a�ecting the under-standing, satisfaction and interaction with the system in di�erent ways. As described inthe in-depth study (see Chapter 5), another strategy when designing for heterogeneous usergroups is to create interfaces that can adapt to the user's individual needs and preferences.This section discusses the value of adding adaptivity to further increase the accessibility ofthe ePRO questionnaire, based on the �ndings of this study.

8.2.1 Value of adding adaptivity

Considering the complications following a stroke are highly individual, a�ecting both physi-cal and cognitive functions with great variations and severity levels, adaptive user interfacescould be an e�ective solution to achieve tailored user interfaces and in that way increaseaccessibility of the system. The �exibility of adaptive user interfaces could also be especiallybene�cial in this target group, since the patients' condition is likely to change over time.

However, applying this strategy to an ePRO questionnaire might not be appropriate,for several reasons. First of all, as mentioned previously, variations in the presentationformat may a�ect interpretation and how questions are being answered [18]. Depending onhow big the di�erences would be between the interface variations, equivalence testing wouldpotentially be necessary to ensure the scores are comparable. However, as mentioned byFindlater and Gajos [53], such evaluation can be di�cult to perform for the same reason.

Secondly, as an ePRO questionnaire may not be used frequently, at least not in this casewhere patients are supposed to report in connection to follow-ups about two times a year,adaptivity features that are based on partial experience with the user, would not be verye�ective, as pointed out by Lavie and Meyer [52]. Of course, detecting and accommodatingfor tapping problems during use could be possible, e.g. by increasing the sensitivity ofbuttons. However, preventing them from happening in the �rst place would be the bettersolution.

8.3. Limitations 39

Although it may be di�cult to gather a su�cient amount information implicitly, i.e. byanalyzing the user's interaction behavior, an ePRO questionnaire still enables the system tobuild rich user pro�les through explicitly retrieved information, assessed from the patient'sreported answers. In a future, more integrated system, other information about the patientcould also be used, such as the patient's medical record. The questionnaire used in thisstudy captures many dimensions of the user's physical and cognitive needs, such as sensoryabilities, hand function and language comprehension, which could be directly used to adaptseveral elements in the user interface. Similar to the MyUI system [54], described in Section5.2.2, font size, amount of text information displayed per screen and whether to activatetext-to-speech functionality are examples of variables that could be adapted based on suchinformation.

However, this information is not known the �rst time a patient answers the questionnaire,and changes in the user interface during use, in this case a question session, could beconfusing [54] depending on how it is applied. An automatic activation of the text-to-speech functionality when a patient reports having trouble reading would probably not be sosmooth. A mixed-initiative strategy, with some level of user involvement in the adaptations,could potentially be a better solution. For example, as in the MyUI system [54], where usersget to decide whether to use a tailored adaptation or not when proposed by the system.However, as further reported in their study, elderly with lower ICT experience might notunderstand the concept of adaptation.

Another way of involving the users is to let them state their preferences themselves, asone of the successfully applied alternatives in the SUPPLE system [41]. Then again, theaddition of too many choices and settings could be di�cult to handle for users in this targetgroup, and should be limited to reduce demands on working memory [32].

In conclusion, adding self-adapting features to systems targeting heterogeneous usergroups, as in this study, could be an e�ective way of increasing accessibility, since theyautomatically tailor to the user's needs without requiring any e�ort from the user. However,its value and applicability in the particular context of an ePRO questionnaire must be furtherinvestigated to ensure the adaptivity does not a�ect how users interpret and respond to thequestions. Additionally, the appropriate level of adaptivity for the task and user group mustbe determined to minimize the usability side e�ects of lost control and unpredictability,possibly requiring some degree of user involvement.

8.3 Limitations

The main limitation of this study concerns the number of users participating in the �nalevaluation. While it falls just within the range of the recommendations (5�10) [18], it isnot enough to provide reliable and statistically signi�cant results that cover the diversity inthis user group. Recruiting potential end users was di�cult within the time and resourcesavailable for this project. For this reason, initial user tests were broadened to include elderlyin general and people with either professional or any personal experience of stroke, regardlessof how recently they were a�ected.

The study also has some validity issues regarding the comparison between the paper- andtablet-based questionnaires. Although no major changes to either instructions, questions norresponse options were made, a fully valid comparison would also require the questionnairesto be equally long. Furthermore, the questionnaires should have been completed in randomorder to avoid bias of the so called `recency e�ect', where users tend to rate recent eventsbetter than earlier events [62]. Unfortunately, this was not possible to do for the �nal

40 Chapter 8. Discussion

evaluation. The time since the patients �lled out the questionnaire on paper might alsohave been too long for the patients to be able to make reliable comparisons.

Another factor a�ecting the validity of the user evaluation results is the fact that thedeveloper of the ePRO system was the person conducting the user tests. This might haveresulted in a tendency in users to rate the system better than what they actually thoughtto satisfy the creator, also referred to as `social desirability bias' [62]. However, the patientswere encouraged to be critical and their observed interaction with the system was alsocarefully considered in the analysis to provide more honest results.

The ecological validity of the study can also be questioned, as the �nal user tests wereconducted in a non-natural environment and usage situation, which might have a�ected thepatients' interaction with the system.

Chapter 9

Conclusions

The tablet-based questionnaire of the developed ePRO system appeared to be both applica-ble and usable for stroke and TIA patients, with the potential of easing use and streamliningthe process of PRO data collection. The �ndings also suggest tapping problems and reduc-ing cognitive load are especially important to consider when designing ePROs for this targetgroup. However, more extensive studies are needed to increase reliability and cover more ofthe diversity seen in this heterogeneous user group.

9.1 Future work

As this study only evaluated a limited part of the thought ePRO system, future versionsshould include log-in functionality to enable storing of user answers and presenting theresults in the Rehab-Compass chart, and ultimately assess its full potential in clinical use.In a more extensive study, it would also be interesting to further investigate the value andapplicability of adding adaptivity features to the system to potentially make it more usableand accessible.

41

42 Chapter 9. Conclusions

Chapter 10

Acknowledgments

First, I would like to thank my supervisors; Fredrik Öhberg at MT-FoU and Lars-ErikJanlert at Umeå University, for their generous support, guidance and insightful discussionsthroughout the development and writing process of this work. I also want to acknowledge myfriends and family for proofreading and providing useful suggestions that helped to improvethe quality of this thesis.

I would further like to express my gratitude to Xiaolei Hu at the department of Re-habilitation Medicine for her great involvement and inspiring enthusiasm, as well as herdoctoral students for their valuable feedback and help. Thanks also to others at MT-FoUfor technical support and input.

Last but not least, a big thank you to all participants of the user tests conducted duringthe development process and �nal evaluation, who kindly shared their stories, experiencesand provided me with invaluable feedback that made this thesis possible.

43

44 Chapter 10. Acknowledgments

References

[1] The Internet Stroke Center: Stroke statistics. http://www.strokecenter.org/

patients/about-stroke/stroke-statistics/ (2017) [Online; accessed 17-Jan-2017].

[2] The Internet Stroke Center: What is a stroke? http://www.strokecenter.org/

patients/about-stroke/what-is-a-stroke/ (2017) [Online; accessed 17-Jan-2017].

[3] May�eld Brain & Spine: Stroke. http://www.mayfieldclinic.com/pe-stroke.htm#(April 2016) [Online; accessed 17-Jan-2017].

[4] World Heart Foundation: Stroke. http://www.world-heart-federation.org/

cardiovascular-health/stroke/ (2017)

[5] United Nations: World population ageing - the united nations. http:

//www.un.org/en/development/desa/population/publications/pdf/ageing/

WPA2015_Report.pdf (2015) [Online; accessed 17-Jan-2017].

[6] Duncan, P.W., Zorowitz, R., Bates, B., Choi, J.Y., Glasberg, J.J., Graham, G.D., Katz,R.C., Lamberty, K., Reker, D.: Management of adult stroke rehabilitation care. Stroke36(9) (2005) e100�e143

[7] Nelson, E.C., Eftimovska, E., Lind, C., Hager, A., Wasson, J.H., Lindblad, S.: Patientreported outcome measures in practice. Bmj 350 (2015) g7818

[8] World Health Organization: ehealth. http://www.who.int/topics/ehealth/en/

(2017) [Online; accessed 19-Jan-2017].

[9] Bennett, A.V., Jensen, R.E., Basch, E.: Electronic patient-reported outcome systemsin oncology clinical practice. CA: a cancer journal for clinicians 62(5) (2012) 336�347

[10] Umeå University: Rehabilitation medicine. http://www.communitymedicine.umu.

se/english/our-units/rehabilitation-medicine/?languageId=1 (2017) [Online;accessed 27-Jan-2017].

[11] Västerbottens Läns Landsting: Medicinsk teknik, forskning och utveckling. https:

//www.vll.se/default.aspx?id=101031 (2017) [Online; accessed 20-Jan-2017].

[12] National Stroke Rehabilitation: What is a tia? http://www.stroke.org/

understand-stroke/what-stroke/what-tiav (2017) [Online; accessed 6-May-2017].

[13] Hu, X., Gustavsson, E., Wester, P., Levi, R., Lindvall, P., Sedeh, A.N., Nyman, K.:Rapid identifying unmet rehabilitation needs with a graphic rehab-compass in the clin-ical practice - a pilot study on patients after subarachnoid haemorrhage. (2015)

45

46 REFERENCES

[14] An, N.: Strategies for overcoming barriers to the implementation of patient-reportedoutcomes measures

[15] Gliklich, R.E., Dreyer, N.A., Leavy, M.B.: Use of patient-reported outcomes in reg-istries. (2014)

[16] Waters, K.: Prioritization using moscow. Agile Planning 12 (2009)

[17] Waracle: Html5 vs hybrid vs native app development. https://waracle.net/

html5-vs-hybrid-vs-native-app-development/ (November 2015) [Online; accessed05-June-2017].

[18] Coons, S.J., Gwaltney, C.J., Hays, R.D., Lundy, J.J., Sloan, J.A., Revicki, D.A.,Lenderking, W.R., Cella, D., Basch, E., et al.: Recommendations on evidence needed tosupport measurement equivalence between electronic and paper-based patient-reportedoutcome (pro) measures: Ispor epro good research practices task force report. Value inHealth 12(4) (2009) 419�429

[19] Eremenco, S., Coons, S.J., Paty, J., Coyne, K., Bennett, A.V., McEntegart, D., et al.:Pro data collection in clinical trials using mixed modes: report of the ispor pro mixedmodes good research practices task force. Value in Health 17(5) (2014) 501�516

[20] Kavalieratos, D., Sullivan, S.M., Hess, R.: Engaging cardiology patients via electronicpatient-reported outcomes: A usability assessment (2014)

[21] Richter, J.G., Becker, A., Koch, T., Nixdorf, M., Willers, R., Monser, R., Schacher,B., Alten, R., Specker, C., Schneider, M.: Self-assessments of patients via tablet pcin routine patient care: comparison with standardised paper questionnaires. Annals ofthe rheumatic diseases 67(12) (2008) 1739�1741

[22] Jensen, R.E., Rothrock, N.E., DeWitt, E.M., Spiegel, B., Tucker, C.A., Crane, H.M.,Forrest, C.B., Patrick, D.L., Fredericksen, R., Shulman, L.M., et al.: The role oftechnical advances in the adoption and integration of patient-reported outcomes inclinical care. Medical care 53(2) (2015) 153

[23] Hess, R., Santucci, A., McTigue, K., Fischer, G., Kapoor, W.: Patient di�culty usingtablet computers to screen in primary care. Journal of general internal medicine 23(4)(2008) 476�480

[24] Gwaltney, C.J., Shields, A.L., Shi�man, S.: Equivalence of electronic and paper-and-pencil administration of patient-reported outcome measures: a meta-analytic review.Value in Health 11(2) (2008) 322�333

[25] Schick-Makaro�, K., Molzahn, A.: Strategies to use tablet computers for collection ofelectronic patient-reported outcomes. Health and quality of life outcomes 13(1) (2015)2

[26] Rodrigues, É., Carreira, M., Gonçalves, D.: Developing a multimodal interface for theelderly. Procedia computer science 27 (2014) 359�368

[27] Caprani, N., O'Connor, N.E., Gurrin, C.: Touch screens for the older user. In: Assistivetechnologies. InTech (2012)

[28] Stroke Association: State of the nation - stroke statistics. https://www.stroke.org.uk/sites/default/files/stroke_statistics_2015.pdf (January 2016)

REFERENCES 47

[29] National Stroke Association: Post-stroke conditions. http://www.stroke.org/

we-can-help/survivors/stroke-recovery/post-stroke-conditions (2017) [On-line; accessed 17-Jan-2017].

[30] Kubis, N.: Non-invasive brain stimulation to enhance post-stroke recovery. Frontiersin Neural Circuits 10 (2016)

[31] Becker, S.A., Webbe, F.M.: Designing for older adult users of handheld technology.In: Engineering in Medicine and Biology Society, 2006. EMBS'06. 28th Annual Inter-national Conference of the IEEE, IEEE (2006) 3297�3300

[32] Kurniawan, S., Zaphiris, P.: Research-derived web design guidelines for older people.In: Proceedings of the 7th international ACM SIGACCESS conference on Computersand accessibility, ACM (2005) 129�135

[33] Loureiro, B., Rodrigues, R.: Design guidelines and design recommendations of multi-touch interfaces for elders. In: The 7th international conference on advances incomputer-human interactions (ACHI 2014). (2014)

[34] Grguri¢, A., Benc, I., De²i¢, S., Mo²mondor, M., Kriºani¢, J., Lazarevski, P.: Design-ing user interfaces for elderly: A case study in applicability of thin vs. fat clients. In:e-Health Networking Applications and Services (Healthcom), 2010 12th IEEE Interna-tional Conference on, IEEE (2010) 99�105

[35] Kobayashi, M., Hiyama, A., Miura, T., Asakawa, C., Hirose, M., Ifukube, T.: Elderlyuser evaluation of mobile touchscreen interactions. In: IFIP Conference on Human-Computer Interaction, Springer (2011) 83�99

[36] Palmblad, M., Tiplady, B.: Electronic diaries and questionnaires: designing user in-terfaces that are easy for all patients to use. Quality of Life research 13(7) (2004)1199�1207

[37] Holzinger, A., Errath, M.: Mobile computer web-application design in medicine: someresearch based guidelines. Universal Access in the Information Society 6(1) (2007)31�41

[38] Smirek, L., Henka, A., Zimmermann, G.: Giving elderly access to smart environments.In: International Conference on Human Aspects of IT for the Aged Population, Springer(2015) 465�475

[39] Story, M.F., Mueller, J.L., Mace, R.L.: The universal design �le: Designing for peopleof all ages and abilities. (1998)

[40] Singh, A., Wesson, J.: Evaluation criteria for assessing the usability of erp systems.In: Proceedings of the 2009 annual research conference of the South African Instituteof Computer Scientists and Information Technologists, ACM (2009) 87�95

[41] Gajos, K.Z., Weld, D.S., Wobbrock, J.O.: Automatically generating personalized userinterfaces with supple. Arti�cial Intelligence 174(12-13) (2010) 910�950

[42] Bunt, A., Conati, C., McGrenere, J.: What role can adaptive support play in anadaptable system? In: Proceedings of the 9th international conference on Intelligentuser interfaces, ACM (2004) 117�124

48 REFERENCES

[43] Norcio, A.F., Stanley, J.: Adaptive human-computer interfaces: A literature surveyand perspective. IEEE Transactions on Systems, Man, and Cybernetics 19(2) (1989)399�408

[44] Yigitbas, E., Sauer, S.: Engineering context-adaptive uis for task-continuous cross-channel applications. In: International Conference on Human-Centred Software Engi-neering, Springer (2016) 281�300

[45] Sundar, S.S., Marathe, S.S.: Personalization versus customization: The importanceof agency, privacy, and power usage. Human Communication Research 36(3) (2010)298�322

[46] Germanakos, P., Mourlas, C., Panayiotou, C., Samaras, G.: Personalization systemsand processes review based on a predetermined user interface categorization. In: Pro-ceedings of the III international conference on communication and reality, digital utopiain the media: From discourses to facts. (2005) 431�444

[47] Akiki, P.A., Bandara, A.K., Yu, Y.: Adaptive model-driven user interface developmentsystems. ACM Computing Surveys (CSUR) 47(1) (2014) 9

[48] Liming Luke Chen, D.R.M., Dr Matthias Steinbauer, P., Castillejo, E., Almeida, A.,López-de Ipiña, D.: Modelling users, context and devices for adaptive user interfacesystems. International Journal of Pervasive Computing and Communications 10(1)(2014) 69�91

[49] The Interaction Desgin Foundation: Context of use (2017) [Online; accessed 25-Jan-2017].

[50] Castillejo, E., Almeida, A., López-de Ipina, D.: User, context and device modeling foradaptive user interface systems. In: International Conference on Ubiquitous Computingand Ambient Intelligence, Springer (2013) 94�101

[51] Ramachandran, K.: Adaptive user interfaces for health care applications. IBM devel-operWorks (2009)

[52] Lavie, T., Meyer, J.: Bene�ts and costs of adaptive user interfaces. InternationalJournal of Human-Computer Studies 68(8) (2010) 508�524

[53] Findlater, L., Gajos, K.Z.: Design space and evaluation challenges of adaptive graphicaluser interfaces. AI Magazine 30(4) (2009) 68

[54] Peissner, M., Häbe, D., Janssen, D., Sellner, T.: Myui: generating accessible userinterfaces from multimodal design patterns. In: Proceedings of the 4th ACM SIGCHIsymposium on Engineering interactive computing systems, ACM (2012) 81�90

[55] Jameson, A.D.: Understanding and dealing with usability side e�ects of intelligentprocessing. AI Magazine 30(4) (2009) 23

[56] Bunt, A., Conati, C., McGrenere, J.: Mixed-initiative interface personalization as acase study in usable ai. AI Magazine 30(4) (2010) 58

[57] Johnston, N., O'Byrne, P., Kolb, M.: The promise of electronic data capture in respi-ratory medicine (2011)

REFERENCES 49

[58] Toepoel, V., Das, M., Van Soest, A.: Design of web questionnaires: The e�ects of thenumber of items per screen. Field Methods 21(2) (2009) 200�213

[59] Toepoel, V., Das, M., Van Soest, A.: E�ects of design in web surveyscomparing trainedand fresh respondents. Public Opinion Quarterly 72(5) (2008) 985

[60] Anthony: Why rounded corners are easier on the eyes. http://uxmovement.com/

thinking/why-rounded-corners-are-easier-on-the-eyes/ (2011) [Online; ac-cessed 15-Mar-2017].

[61] Bryant, K.: Rounded corners and why they are here to stay. https://designmodo.

com/rounded-corners/#comments (2012) [Online; accessed 15-Mar-2017].

[62] Natesan, D., Walker, M., Clark, S.: Cognitive bias in usability testing. In: Proceedingsof the International Symposium on Human Factors and Ergonomics in Health Care.Volume 5., SAGE Publications (2016) 86�88

Designing for diversity: A tablet-based ePRO system to ... · Designing for diversity: A...

Documents

Transcript of Designing for diversity: A tablet-based ePRO system to ... · Designing for diversity: A...