Towards Standards in User Interface Design of Networked Devices

A Thesis Presented by

Sanel Selimovic

to

The Department of Mechanical and Industrial Engineering

in partial fulfillment of the requirements for the degree of

Master of Science

in

Industrial Engineering

in the field of

Human Machine Interaction

Northeastern University Boston, Massachusetts

December 2010

II

Abstract Networked appliances have long held promise of making people’s lives easier by

allowing users complete access over their functionality through an intermediate device such as a

tablet computer. Corporations often see no benefits to standardizing user interface across their

multiple appliances, or they feel that their proprietary software offers them a competitive edge in

the marketplace.

This thesis explores the underlying set of principles for home appliances. It attempts to

identify user expectations in terms of what they want to see in their networked appliances in

terms of features and functionality, but also provide guidelines for future developers. To

accomplish this, two prototypes were developed and tested using a System Usability Scale (SUS)

survey and participant success rates were measured in terms of the time it took to complete each

task, and the success rate across participants and across tasks. The two experiments were

conducted using a remote usability tool. Study results indicate statistically significant differences

between the two prototypes.

III

Acknowledgements I would like to thank everyone who has helped me out on this journey. First and foremost I

would like to thank my advisor Yingzi Lin for all of her trust, help and support she has given me

on my thesis work and in my academic work. I would also like to thank my friend and statistical

guru Ahamed Altaboli for all of his help on this and many other projects. Thank you to everyone

else whom I haven’t specifically called out.

My time in Boston, and my time at Northeastern in our lab, has been the most amazing two years

of my life. Once again, thank you Professor Lin for giving me this opportunity.

Sincerely,

Sanel Selimovic

IV

Table of Contents 

Chapter 1: Introduction .......................................................................................................... 1 Overview ...................................................................................................................................................... 1 Background .................................................................................................................................................. 2 

Motivation ............................................................................................................................................... 2 Simplified UI Methods ............................................................................................................................. 4 

Research Problems ...................................................................................................................................... 5 

Chapter 2: Conceptual Framework .......................................................................................... 6 Approaches to Simplified UI ........................................................................................................................ 6 

Simplifying Consumer Electronics Interfaces .......................................................................................... 7 Mixed Initiative (Collaborative) Interfaces .............................................................................................. 9 Goal‐Oriented Interfaces ....................................................................................................................... 10 Other Interaction Modes and Guiding Principles .................................................................................. 11 

System Usability Scale ............................................................................................................................... 12 Error Rates ................................................................................................................................................. 13 Task Completion Times .............................................................................................................................. 14 

Chapter 3: Experiment Design ............................................................................................... 14 Hypothesis ................................................................................................................................................. 14 

Definition ............................................................................................................................................... 15 Performance Data ................................................................................................................................. 15 Task Completion and Error Rates .......................................................................................................... 16 

Experiment Setup ...................................................................................................................................... 16 Introduction .......................................................................................................................................... 16 Initial Survey .......................................................................................................................................... 17 Prototype 1 Setup ................................................................................................................................. 19 Prototype 2 Setup ................................................................................................................................. 25 Scoring the System Usability Scale (SUS) .............................................................................................. 30 

Experiment Procedure ............................................................................................................................... 32 Task List ................................................................................................................................................. 35 

Chapter 4: Experiment Results .............................................................................................. 36 Initial Survey Results .................................................................................................................................. 36 System Usability Scale (SUS) Results ......................................................................................................... 43 

SUS Results Analysis .............................................................................................................................. 43 Performance Data Results ......................................................................................................................... 46 

General Results ..................................................................................................................................... 46 Statistical Analysis ................................................................................................................................. 53 Overall Summary ................................................................................................................................... 56 

Chapter 5: Conclusion ........................................................................................................... 56 

Appendix .............................................................................................................................. 58 Appendix A: SUS Survey ............................................................................................................................. 58 Appendix B: Instructions to Participants ................................................................................................... 59 

References ............................................................................................................................ 60 

V

Table Index Table 1: Summary Of Participant Interest In Remote Control Of Given Appliances .................................. 18 

Table 2: List Of Tasks And Order They Were Presented To The User ........................................................ 35 

Table 3: Breakdown Of Participant Responses To Short Survey ................................................................ 37 

Table 4: Most Frequently Requested TV Features And Functionality ....................................................... 39 

Table 5: Summary Of Responses For The A/C ........................................................................................... 40 

Table 6: Summary Of Responses For Heater ............................................................................................. 40 

Table 7: Summary Of Responses For Lights ............................................................................................... 41 

Table 8: Summary Of Responses For DVD Player ...................................................................................... 41 

Table 9: Summary Of Responses For Surround Sound .............................................................................. 42 

Table 10: Summary Of Responses For Home Security ............................................................................... 42 

Table 11: Summary Of Responses For Door Locks ..................................................................................... 42 

Table 12: Prototype One ‐ Summary Of Responses To The SUS Survey .................................................... 44 

Table 13: Prototype One ‐ Summary Of Statistical Data For SUS Survey ................................................... 44 

Table 14: Prototype Two ‐ Summary Of Responses To The SUS Survey .................................................... 45 

Table 15: Prototype Two ‐ Summary Of Statistical Data For SUS Survey .................................................. 45 

Table 16: Average Task Completion Time ................................................................................................. 46 

Table 17: Average Number Of Errors Across Participants And Across Tasks For Experiment 1 ................ 48 

Table 18: Average Number Of Errors Across Participants And Across Tasks For Experiment 2 ................ 50 

Table 19: Experiment 1 – Mean Task Ease Of Use Ratings ........................................................................ 51 

Table 20: Experiment 2 – Mean Task Ease Of Use Ratings ........................................................................ 52 

Table 21: Average Number Of Seconds To Complete A Task .................................................................... 53 

Table 22: Unpaired T‐Test Of Average Completion Time In Seconds ........................................................ 54 

Table 23: Success Rate Across Tasks .......................................................................................................... 54 

Table 24: Unpaired T‐Test Of Success Rate Across Tasks .......................................................................... 55 

Table 25: Number Of Errors Across Participants ....................................................................................... 55 

Table 26: Unpaired T‐Test Of Task Completion Across Participants .......................................................... 55 

VI

Figure Index Figure 1: Sample Screen Of Diamondhelp Application Developed For Collagen (Rich et al, 2005) ........... 10 

Figure 2: A Comparison Of Mean System Usability Scale (SUS) Scores By Quartile, Adjective Ratings, And 

The Acceptability Of The Overall SUS Score (Bangor et al., 2008) .................................................... 13 

Figure 3: Completed Application Demo ..................................................................................................... 20 

Figure 4: Prototype One ‐ TV Control Panel .............................................................................................. 21 

Figure 5: Prototype One ‐ A/C & Heat Control Panel ................................................................................. 21 

Figure 6: Prototype One ‐ Home Lights Control Panel ............................................................................... 22 

Figure 7: Prototype One ‐ DVD Player Control Panel ................................................................................. 23 

Figure 8: Prototype One ‐ Surround Sound Control Panel ......................................................................... 23 

Figure 9: Prototype One ‐ Alarm Clock Control Panel ............................................................................... 24 

Figure 10: Prototype One ‐ Home Security Control Panel ......................................................................... 24 

Figure 11: Prototype Two ‐ Revised TV Control Panel ............................................................................... 25 

Figure 12: Prototype Two ‐ Revised A/C & Heat Control Panel ................................................................. 26 

Figure 13: Prototype Two ‐ Revised Home Lights Control Panel ............................................................... 27 

Figure 14: Prototype Two ‐ Revised DVD Player Control Panel ................................................................. 28 

Figure 15: Prototype Two ‐ Revised Surround Sound Control Panel ......................................................... 28 

Figure 16: Prototype Two ‐ Revised Alarm Clock Control Panel ................................................................ 29 

Figure 17: Prototype Two ‐ Revised Home Security Control Panel ............................................................ 29 

Figure 18: The SUS Survey Used In This Research (In Its Original Form) ................................................... 31 

Figure 19: Instructions On How To Use Loop11 ........................................................................................ 34 

Figure 20: Average Task Completion Time In Seconds Across Tasks ......................................................... 47 

Figure 21: Successful Task Completion Across Participants ...................................................................... 50 

Figure 22: Successful Task Completion Across Participants ...................................................................... 51 

Figure 23: Mean Ease Of Task Rating, Higher Values Are Better ............................................................... 52 

1

Chapter 1: Introduction  

Overview 

Networked appliances and devices have long held a promise of making it easier for

people to manage their time, lives and energy use. Unfortunately development in this area has

been limited and has found little acceptance in the consumer market outside the hands of a few

knowledgeable power-users. Many consumer electronics companies have attempted to create an

ecosystem of networked and operational home appliances.

The focus of this thesis is on developing a standardized user interface with respect to

functionality and design of total remote control of home appliances and devices by focusing on

three widely-accepted usability measures: effectiveness, efficiency and satisfaction.

Effectiveness and efficiency provide a quantitative measure of success as users interact with the

interface and satisfaction scale provides a qualitative overview of user satisfaction with the

presented system.

Twenty Internet users participated in a remote usability test. Their error rates, satisfaction

and speed of task completion were measured. The three usability measures demonstrated

statistically significant improvements in overall interface design, with a final SUS score of 79%,

the p value for the difference in average time to complete a task between two prototype at p =

.0323, the number of errors across tasks at p = .0206 and number of errors across participants at

p = .0287. This research serves as a basis for building user interface standards by considering end

users goals in terms of functionality users want and expect from their appliances.

2

Background 

Motivation 

The importance of standardizing user interfaces and considering human factors in product

and system design is impossible to overstate. This was most evident at the Three Miles Island

accident where a series of operator errors led to a catastrophic failure of a nuclear power plant

and a partial meltdown of the nuclear core (Meshkati, 1991). As a consequence of the accident,

the government implemented many changes, including standardization of instruments and

controls for operating the plant (Backgrounder on the Three Mile Island Accident, 2009)

Although standardization has often occurred as a result of accidents or as a preventative

measure to stop accidents before they happen, standardization is happening in other areas that

carry fewer real-world dangers. In areas, such as computer science, the sophistication level of

machines and software available on these machines has happened mainly in part due to these

standardized interfaces (Myers, Hudson & Pausch, 2000). This is most noticeable when

considering user interfaces across many different platforms like Windows, Macintosh and Unix.

A single application such as an Internet browser tends to offer similar functionality and same

functional layout across various platforms. Many of the graphical interface methods were

originally developed by Xerox, and adopted on Macintosh in 1984 (Reimer, 2005) and later

found its way to other platforms such as Windows and Linux. The standardization of Internet

browsers in terms of features, functionality and their look and feel has played a critical role in

shaping the way these technologies have evolved over the past 20 years.

A standardized user interface offers many benefits to end-users. Among these benefits

are: overall increase in user productivity, decrease of stress and frustration, improved

3

interoperability among different devices types such as cell phones to tablets to PC’s and an

improved overall product that is satisfying to use and aesthetically pleasant (Reed, et al., 1999).

Despite of standardizing attempts of UI across personal computers, this standardization

does not translate well onto pocket and portable devices such as smart phones and tablets (Lasky

et al. 1998). Pier and Landay (1992) found that pull-down menus on Windows widgets do not

work well on wall-size displays because they presume a certain height requirement of their users

meaning that not everyone would be able to reach the pull-down menu. In today’s modern

devices which include Apple iPhone, Microsoft Windows Phone 7 and Google Android the UI

across all three major competitors is completely different. Android devices suffer from even

greater fragmentation (Liau Yun Qing, 2010) not only in respect to the number of different

operating system iterations available, but also the variety of non-standard user interfaces

manufacturers are allowed to place on top of the underlying Android operating system.

There have been several promising attempts to bring about a universal interface for

networked home appliances (Rich, et al, 2005). However, their parent corporations have either

canceled these projects prematurely or the research team has moved on to address different

research problems. The reason for a lack of a universal home interface across devices occurs for

several reasons. In many cases it is not feasible for a large corporation to develop a standardized

UI across multiple devices. Companies such as General Electric (GE) develop not only consumer

level products such as phones, microwaves and washing machines but advanced military

equipment and industrial machinery. For a company as large as GE, for instance, it may not be

possible or worthwhile to standardize the UI between a microwave, a washing machine and a jet

fighter. The second reason for a lack of industry-wide standardization across devices is because

4

companies often feel that having their own uniquely identifying UI offers benefits and a

competitive advantage (Rich, et al, 2005).

Developing a consistent and a usable system has long taken a back seat as most

manufacturers focus on price-sensitive consumers and develop their products for the lowest

common denominator, often sacrificing user experience research and design along the way in

order to gain competitive advantage by being the first to deliver a product to the marketplace.

One company that has excelled at maintaining stringent user experience standards across

its devices and has made a business model out of standardized usability is Apple. With the

millions of devices Apple has sold (Apple Inc, 2010), the company has demonstrated that there is

a demand for usability, for consistent user experience, user interface, and that this can be a

highly profitable business model.

The technological march towards faster and more powerful devices has led to increased

complexity. In fact, as consumer devices have grown more complex and rich in features and

functionality, they have in turn become overly complex and difficult to use (Brouwer-Janse et

al., 1992). In response to the growing complexity of these devices researchers have developed

several methods for approaching with this issue.

Simplified UI Methods  

This research project is focused on developing user interface standards that would

provide guidance on best layouts and most common features and functionalities of networked

devices. The interface could be loaded on to a tablet device such as an iPad and allow remote

control of an array of networked appliances. At present time, almost all research efforts are

devoted to developing the underlying communication technology while little effort has gone into

5

developing the terminology and a look and feel that remains consistent across multiple devices

such as Android-, iOS- and Windows-based phones and tablet machines.

There are two parts to the current usability crisis (Rich et al, 2005), complexity and

inconsistency. The first part of it is that the proliferation of computers and computer technology

has made it too easy for manufacturers to add features. These features often exceed the capacity

of current UI. The second part is that there is often little to no consistency between devices that

offer similar or same functionality and between the devices coming from the same manufacturer.

Before a user even begins to use a device he or she starts to form a conceptual model of

how a device is going to work (Norman, 2007). Without a conceptual model it becomes difficult

and in some cases impossible to predict the consequences of our actions. This may work well

when the interface is responding a predictable manner; however, as soon as errors are introduced

into the system it breaks down.

The problem with current approaches to home networked appliances and interfaces is that

all projects focus exclusively on the underlying technology and very little attention is paid to the

overall look and feel of the interface. This research explores the underlying set of principles for

home appliances. It attempts to identify user expectations in terms of what they want to see in

their networked appliances in terms of features and functionality, but also provide guidelines for

future developers.

Research Problems  

The goals and methods of this research are not the same as those employed by previous

studies. Previous studies looked at networked appliances from a programming point of view.

Other studies focus more on the theory of developing an interface rather than an iterative process

6

of developing an interface. This study, in contrast, focuses on finding out the functionality that

users actually want from their devices, rather than focusing on low-level code that will run below

the surface and not be accessible to users.

Chapter 2: Conceptual Framework 

Approaches to Simplified UI 

As technology continues to progress at a rapid pace we will find that many of the

products around us will come equipped with computational capabilities and wireless receivers

[Nakajima et al. (2002), Gershenfeld (2000), Gellersen et al. (2000), Kindberg (2000), Hopper

(1999), and Want et al (1995)]. Ultimately, this relentless technological progress will lead to

every device in the world being connected to one another through the Internet (Nakajima et al.,

2002). Devices in the future will not all be a part of a single homogeneous network. Instead they

will run on many different processors and will use different means to provide users with input

and output functionality. It is therefore imperative that this heterogeneity be taken into account

when designing for future interfaces.

In 2005, 43.2 million US households had an Internet broadband connection and 30

million had a personal home network (Wang, 2006, Shehan & Edwards, 2007). Frequently, a

network consists not just of a single device but also of a multitude of devices all interconnected

with one another (Wang, 2006). And although it appears that an average household has an

advanced network set up already and that there aren’t any issues with it, return rates tell a

different story. The return rate for home networking devices is 20-30% and constitutes the most

frequently returned item at large electronics retailers (Scherf, 2002). A user study indicates that

the primary reason for such a large return rate is overall technical complexity in setting up a

7

personal home network (Laszlo et al, 2002). All of these problems can be traced back to the

inception of the Internet. The people who developed the Internet were technically sophisticated

so that the usability and ease of use of networks, networking devices and products was not a

consideration on their minds (Geeksquad, 2010). Research seems to suggest that networked

appliances are suffering from substantial usability crisis and need to be addressed urgently

(Grinter et al., 2005 & Kiesler et al., 2000).

There are several overarching ideas in the area of networked appliances. A vast number

of networked appliances can be classified under one or more of these categories. These big ideas

can be classified into three different approaches to developing smart home appliances. First

approach to smart home development is to simplify consumer electronics interfaces

(Alexandersson, Zimmermann and Bund, 2009; Lieberman and Espinosa, 2006), thereby

allowing each device to be context sensitive and allow for user debugging. Second approach is

often referred to as mixed-initiative (collaborative) interfaces (Rich et al, 2005) whose purpose is

to cooperate with users to satisfy their goals. Third group belongs to the category of goal-

oriented interfaces. This type of device attempts to infer user’s goals based on their actions. The

last method applied to smart home devices is that of self-explanatory interfaces (Lieberman and

Espinosa, 2006). These three approaches to simplifying the UI are documented in the following

sections.

Simplifying Consumer Electronics Interfaces  

This first approach of current UI methods can be broken down into infra-red remote

controls, custom installation homes and media centers (Alexandersson, Zimmermann and Bund,

2009). Infra-red remote controls are the most common and well-known interface devices. These

devices are limited in functionality and features because infra-red technology is a one-way

8

communication street and the user can only provide input to the television for instance, but they

cannot receive feedback through the infra-red remote control. The Harmony Remote from

Logitech is only capable of transmitting information one way, however, it is not capable of

providing the user with feedback on things such as what radio station they are tuned into, or

appliance information such as whether the TV is on or off. This makes its functionality limited.

A proposed solution to this has been the development of a universal remote control (URC),

however, the problem with URC’s is that they do not have an ability to control devices beyond

those originally programmed into them because they do not have any means of upgrading.

The custom-made home category of available user interfaces is represented by an array of

appliances that has been specifically designated to provide a total home control to persons with

disabilities and older users. This approach to home control offers substantial benefits to the end

user, however because each setup is custom-tailored to each home the system costs can quickly

become unaffordable to most target users. Multiple projects at MIT’s house_N, Georgia Tech

University’s Aware Home, and others attempt to address the problems of smart home UI.

However, the focus of these projects is on home sensors rather than the UI itself (Lieberman and

Espinosa, 2006).

The third category is a media center category. It is made up of computer-TV hybrids that

extend the computer functionality by delivering user’s media to the TV via the computer. Media

centers such as Windows Media Center pull together the functionality and features of multiple

devices: a TV tuner card, a digital video recorder (DVR), they provide access to on-demand

content, internet streaming video and audio, a surround sound system and a photo viewer. There

are several limitations to this approach. One limitation is that media centers provide little to no

accessibility to persons with disabilities. The second drawback of media centers is that they do

9

not currently offer alternative interface modalities such as gesture control and do not offer a way

to interact with the interface besides using a PC (Alexandersson, Zimmermann and Bund, 2009).

Mixed Initiative (Collaborative) Interfaces  

The dominant method of user interaction at the moment is direct manipulation (Rich et al,

2005). Direct manipulation is easy to implement into interfaces that lend themselves to natural

interaction such as pointing, clicking and dragging. As a result of the difficulty in applying these

interaction paradigms into broader Collaborative, or mixed-initiative, interface is based on

Collagen (Rich et al, 2005). Collagen is an “application independent collaboration manager” (pp.

2). This software is placed between the user and the machine and acts as a dialog manager. Prime

example of a Collagen-based interface is DiamondHelp software developed by Mitsubishi (Rich

et al, 2005). DiamondHelp is used for building GUI, it is based on reusable Java framework. This

approach assumes that technology is difficult for humans to use and that only a small fraction of

that functionality. Authors of this study felt that there are two principal reasons why people find

technology difficult to use. Two reasons for this difficulty are because interaction does not

follow any conventions and second reason being inconsistency in user interface design. Authors

of this study note that the system at its present state does not have a standardized UI including

things such as a color pallet, widgets etc. They considered this to be beyond the scope of their

research.

10

Figure 1: Sample Screen Of Diamondhelp Application Developed For Collagen (Rich et al, 2005)

Goal‐Oriented Interfaces 

The third approach is that of goal-oriented and commonsense interfaces. These types of

interfaces infer goals from concrete actions and reformulate search queries (Lieberman and

Espinosa, 2006). An example of this would be Roadie. The system uses an OpenMind

Commonsense knowledgebase (Singh, 2002) through a Graphlan (Blum and Furst, 1997)

implementation. Based on the task that a user is attempting to accomplish, the interface will

provide explanations or suggestions for other goals that a user may want to accomplish. For

instance, if a user inserts a DVD into their computer then the interface would provide several

options such as watching a movie on DVD, recording a DVD movie, listening to music on the

DVD etc. (Lieberman and Espinosa, 2006). Roadie can only work with devices that provide a

method for controlling their functions and the devices must be able to query their own state. The

ability of the device to query its own state is what allows Roadie to monitor what is happening

and to interpret that information as user’s actions. They conclude that devices should reduce their

11

own complexity. They should sense, anticipate, and respond to activities at home (Davidoff et al,

2006 & Lieberman and Espinosa, 2006)

Other Interaction Modes and Guiding Principles 

The currently dominant human-computer interaction paradigm is direct manipulation

(Shneiderman and Plaisant, 2005). This method of interaction is common to most modern

devices and includes things such as pointing, dragging, sliding etc. The main idea behind direct-

manipulation interfaces is that the objects in a user interface are visible, that the actions can be

undone, the speed with which tasks can be accomplished is fast and instead of using a command

line the user can use direct manipulation through the mouse to move objects and change their

states on the screen (Shneiderman and Plaisant, 2005). These ideas epitomize directly-

manipulate interfaces and have been extended to other forms of interaction such as virtual

reality, augmented reality and tangible user interfaces. According to Shnederman and Plaisant

(2005), direct-manipulation is beginning to emerge as a way manipulate and automate home

appliances.

Donald Norman (2007) outlines seven principles that engineers should adhere to when

developing interaction rules for our machines. The first rule is that designs should be close

approximation of people’s natural interactions and motions they use to interact with one another.

The required input into the machine and the obtained output should be predictable. A designer

needs to provide a good conceptual model as to the workings of the machine. Fourth design rule

is that the output a person gets should be understandable to its users. As part of the natural

interaction a system should be continually aware of itself and it’s own state of being. This

awareness should not hinder its use and should not be so annoying to the user that it is

completely distracting from its primary purpose. The last design rule emphasizes the need for

12

natural mappings in order to aid with the interaction. The rule of natural mappings is a method of

describing a relationship between a control, or an action that a person takes upon the interface

and the way the system reacts. To naturally map something means that the interface corresponds

to similar physical actions and cultural norms the users experience (Norman, p. 23). For

example, turning a steering wheel left logically maps to a car turning left.

System Usability Scale 

One of the easiest ways to determine overall usability of a system is by administering a

System Usability Scale (SUS) survey to the participants. The survey provides researchers with an

idea of the overall usability of any given system as a percentage. John Brook developed the SUS

survey (Brooke, 1996). The purpose of this survey, as he put it, was to provide a “quick and dirty

usability scale” that could easily be picked up by any researcher to derive a quick overall

usability rating.

According to Bangor et al. (2008) the success of SUS is attributable to several reasons.

The mean reason, they say, it the SUS survey’s indifference to the platform being tested. The

survey is not tied to any single system, which means it lends itself to use on many different

platforms. The second reason, they state, is the ease of use of the survey. This makes SUS a very

understandable survey to participants and to administrators. Another reason for the proliferation

and success of SUS survey is its ability to provide a single score, a percentage out of one

hundred, which can be easily understood by anyone on the team. Lastly, unlike many other

proprietary and expensive surveys, the SUS survey was made free and available to general

public.

Bangor et al (2008) undertook the most recent empirical evaluation of the SUS survey.

Their study analyzed 2300 SUS surveys over the prior 10 years and their results indicate 85%

13

reliability compared to other usability surveys. To make full sense of results they provided an

overall scoring guide (see Figure 2).

Figure 2: A Comparison Of Mean System Usability Scale (SUS) Scores By Quartile, Adjective Ratings, And The Acceptability Of The Overall SUS Score (Bangor et al., 2008)

The SUS survey is presented to the participants upon completion of their task. The survey

is presented in its original unmodified form. After completing an unmodified SUS survey, the

participants are given a second set of questions soliciting free-form feedback and demographic

information. This information is not critical to the study but it may be useful in discerning any

confounding variables in the study.

Error Rates 

Error rates are some of the most important kind of measure in usability studies (Albert,

Tullis, Tedesco, 2010). Errors are defined as a user’s inability to complete a given task or a belief

that they had completed a task when they have not indeed done so. Error rate can be measured in

multiple ways, but it is most frequently a binary measurement. It means that participants either

completed the task or they did not complete a task. Successfully completed tasks are given a one

and unsuccessful or failed tasks are assigned a zero. The software that is used for this remote

usability research does not provide an error rate with the current prototype. This is because the

application is intended for use with web sites rather than applications. The remote usability

14

application, however, does offer the ability to record participant interaction with the testing

prototype. By watching recorded participant sessions the error rates are collected and each task is

marked as complete or incomplete.

This experiment considers two types of error rates: (1) the error rates committed by

individual participants across all eleven tasks and (2) the error rates across all participants for

every given task. This will provide an overall picture of number of errors per participant and

number of errors per task.

Task Completion Times 

Task completion times have long been used for usability measurements and are

frequently viewed to be as important as error rates (Albert, Tullis, Tedesco, 2010). Data on task

completion times can be presented in two ways: (1) for all tasks or (2) only the correct tasks.

This study will report both because the information it provides is valuable. Task completion

times are measured by the remote usability testing application Loop11. A task is considered to

have started once a participant loads the application and a task is considered complete once a

participant pushes the “Task Complete” button at the top of the page frame. Time it took each

participant to complete a task is then automatically tabulated and presented.

Chapter 3: Experiment Design 

Hypothesis 

This section defines the main hypotheses under investigation by this experiment. The

experiment consists three measurements: effectiveness, efficiency, and satisfaction. Data on

15

effectiveness and efficiency are collected through remote usability application provided by

Loop11 service. The last measurement of satisfaction is collected using a SUS score.

Definition 

The main goal of this experiment is to build an optimal interface that matches users’

expectations in terms of functionality and usability. This study assesses results according to

effectiveness, efficiency and user satisfaction of the interface. Effectiveness of a design is

defined as the number of errors committed by each participant in every task. Efficiency of a

design is defined as the amount of time it takes each participant to complete a given task. Lastly,

the satisfaction measure is defined as an overall percentage score of user’s satisfaction with the

system.

Performance Data 

Three main variables are under investigation by this study, these include: effectiveness,

efficiency and satisfaction. Effectiveness of a design is measured by the number of errors

produced by each design, efficiency measures the average time it takes each participant to

complete a task, and satisfaction is a qualitative measurement intended to gauge overall usability

of a system. A remote usability testing application will collect all data on user interaction. The

hypotheses for each of these variables are:

1. Participants of experiment two should perform fewer errors than participants in

experiment one.

2. Participants of experiment two should perform faster than participants in experiment one.

3. The qualitative measurement of overall system usability satisfaction should be higher in

experiment two than in experiment one.

16

Task Completion and Error Rates 

This research considers two types of performance data: average task completion time and

error rates between tasks and participants. Error rates are frequently displayed as binary data and

this research will do the same. The average task completion time is measured in seconds and

milliseconds. The System Usability Scale (SUS) score is used to provide an assessment of the

overall system usability and any improvements that occur through prototyping The SUS score is

reported as a percentage. It is intended to present a quick and easy way of comparing qualitative

results between the two prototypes.

Experiment Setup 

Introduction 

This research will test two iterations of a prototype. The research will take place online in

an automated remote usability study through a service Loop11. This service allows researchers to

create tasks, administer pre and post task questionnaires and provides measures of response times

for every participant and their task completion or failure rates. The study was conducted as an

online remote usability study due to lack of access to proper usability testing facilities at the

researcher’s abode and the geographical disparity between the main research office. Remote

usability testing offers several benefits when compared to standard live testing. Remote testing

allows researchers to test anywhere, to get a geographic and demographic diversity they could

not in the lab (Bolt & Tulathimutte, 2010). This experiment is approved by and conducted under

supervision of the principal investigator.

17

Initial Survey 

An initial survey was developed before any prototyping or experiments took place. The

purpose of this survey was to gauge people’s interest in remote controlled appliances, which

appliances people would like to control remotely and what functionality they are most likely to

wan to want their remote controlled appliances to have.

Twenty-one people participated in the initial survey. The people were recruited through

online social media web sites such as Twitter and Facebook and through Craigslist. The

questionnaire was developed using Google Documents form. It allows researchers to create an

online survey and uses it to tabulate people’s responses to the questionnaire. Participants were

asked to complete a total of 27 questions. The first question listed the most frequently found

appliances in a home. The participants were asked, “What devices or appliances would you like

to be able to control remotely? Check all that apply.” Table 1 summarizes participant responses

to this question.

18

Table 1: Summary Of Participant Interest In Remote Control Of Given Appliances

  Device Name 1  TV 2  Air Conditioning 3  Lights 4  DVD Player 5  Heater 6  Surround Sound 7  Alarm Clock 8  Home Security System 9  Door Locks 

10  Stereo 11  Clothes Washer 12  Dishwasher 13  Computer 14  Clothes Dryer 15  Stove 16  Microwave 17  Fridge 18  Other 

After completing this question all participants were asked about detailed functionality of

each of the devices. Subsequent questions for each of the devices followed the following format:

“Which of the following actions would you most likely want to perform remotely on your (insert

device name here)” The participant was then given a list of the most common functionality for

each of the devices. They could check as many of the features of each device as they wanted.

They could mark none or other and write in their own response as an option also.

In total, twenty-one participants completely filled in the survey. This survey was used as

a basis for creating the two following prototypes. Results of this survey can be found in

Experiment Results chapter of this research under “Initial Survey Results”.

19

Prototype 1 Setup 

The first prototype was developed based on initial feedback from the user survey. The

survey solicited input on two main areas: (1) the devices a user is likely to want to control

remotely through a tablet-like interface, and (2) the granularity of control users would like to

have over those devices. Axure Rapid Prototyping (RP) application was used develop a high

fidelity functional application mockup. Axure RP is used by user experience professionals in the

field and allows for rapid prototyping with varying degrees of fidelity. Based on the results from

the initial survey eight devices and their full functionality were implemented for the initial

mockup.

Figure 3 is a screenshot of a completed application demo. The application consists of

seven main devices: television, air conditioning & heat, home lights, DVD player, surround

sound, alarm clock and home security. Although the initial user survey listed Air Conditioning &

Heater as two separate devices, for the purposes of this application the two were combined into

one menu. This was done primarily because the two devices offer very similar or same

functionality and because in many modern homes with an air conditioner, the same control panel

is used to set the cold or heat index in a home.

20

Figure 3: Completed Application Demo

All buttons are laid out on the left-hand navigation and provide an illusion that they are

connected to the main menu display, which is framed by a black border. The two additional

buttons found in the prototype are “Switch User” button and “Add new device…” button. These

buttons provide no functionality in the current demo but were added as placeholders for future

functionality.

Pushing on the button labeled “TV” brings up the TV control panel (see Figure 4). The

TV control panel consists of the on/off component, channel keypad, volume control and buttons

for basic setup. In this application demo all buttons respond to pushes by turning yellow and

timing out after several seconds to denote they have been pushed. Pushing the “Turn On” button

on any of the control panel changes the button label from a red “Off” icon to a green “On” icon

on the device button.

21

Figure 4: Prototype One - TV Control Panel

There are two ways for users to know their current location in the application. The

primary way is the device name inside each control panel. The secondary way is by changing the

button border to a dark black and making the button itself darker. These two visual cues are a

way of telling the user where they’re at within the interface.

The A/C and Heat control panel is simple (see Figure 5). It is made up of an on/off

component and a method for controlling home temperature and humidity. The system also allows

users to select an auto-off time.

Figure 5: Prototype One - A/C & Heat Control Panel

22

The Home Lights control panel is also straightforward (see Figure 6). The panel is broken

up into three sections. There is a global light control referring to “All lights” and it allows users

to turn on or off all lights in a house. Second portion is granular and allows users to see

individual status of lights throughout the house. The last section of the control panel is the “Add

new room…” button, which allows users to add light control to additional areas of their homes.

Figure 6: Prototype One - Home Lights Control Panel

The DVD Player control panel was a significant challenge because users requested total

control of their DVD player through this interface (see Figure 7). Only some functionality of a

DVD player was implemented. This was done due to lack of standards between manufacturers

and different model numbers of DVD players, since each one offered different functionality and

capabilities. A total control of all of DVD Player’s functionality would be difficult to replicate in

an application.

23

Figure 7: Prototype One - DVD Player Control Panel

Surround sound (see Figure 8) was another challenge because of different modes and

functionality each receiver box offers. Two main functionalities, however, are present in all

surround sound receiver boxes. These are: the ability switch various audio and video input

modes instantly and the ability to change volume.

Figure 8: Prototype One - Surround Sound Control Panel

24

Alarm clock control panel (see Figure 9) was straightforward because an alarm needed to

offer only one basic feature and that is the ability for users to select hours and minutes.

Figure 9: Prototype One - Alarm Clock Control Panel

Lastly, Home Security control panel (see Figure 10) is split into three sections like the

lights control panel. There is a global control of a security system, which allows users to arm and

disarm the home security alarm. The second section offers more granular view and control of

locks around the house. In this case controls for Front and Back door are only provided.

Figure 10: Prototype One - Home Security Control Panel

25

Prototype 2 Setup 

Based on Experiment 1 and feedback received from users on Prototype 1, a second

Prototype was developed. Second prototype was build around the idea that common functionality

should be grouped into steps and the entire system took on a more wizard-like user interface.

Sections were visual separated and logically grouped, both within each control panel and across

different control panels.

The most obvious immediate difference between Prototype 1 and Prototype 2 is that TV

and Home Lights were left as “On” by default. Jumping into the TV control panel we are

presented with a different user interface (see Figure 11).

Figure 11: Prototype Two - Revised TV Control Panel

The first step to the control panel of every device now is the ability to turn the device on

or off. The reason this functionality was called out in step one is because in the Prototype 1,

users were allowed to click anywhere in the demo and they were not sure how to proceed or what

steps to take and in which order. Often time users would change the channel, turn up volume and

26

lastly turn on the TV itself. Each section was separated by a pleasant blue hue and each section

has a brief header describing what actions to take.

Based on users interaction with Prototype 1 two new components were added: (1) an

output to component and (2) status of device component. The “Output to” component allows

users to select where they would prefer to hear the sound coming from, the TV or the Surround

Sound. This allows them to skip the secondary step of going into the Surround Sound control

panel and having to select and figure out which Audio/Video input corresponds to the television.

A status component was added to all the devices where it made sense. The status component

gives an overview of the current state of a device, in this case the television. It lets the user know

that the TV is on, what the current channel is set to, and that sound is being sent out through TV

speakers rather than surround sound.

For A/C & Heat (see Figure 12) no functionality was added, but the ability for the A/C to

turn off automatically was moved to the right hand side into Step 3.

Figure 12: Prototype Two - Revised A/C & Heat Control Panel

27

Home lights (see Figure 13) benefited from visual separation and by having the dark grey

background of each component changed to white. Compared to Figure 6 each background panel

is now clearer, contrasts well with the background and it is immediately known what

functionality goes with what.

Figure 13: Prototype Two - Revised Home Lights Control Panel

The DVD player control panel also received substantial changes (see Figure 14). Based

on users interaction with Prototype 1, two new components were added: (1) an output to

component and (2) status of device component. The “Output to” component allows users to

select where they would prefer to hear the sound coming from, the TV or the Surround Sound.

This allows them to skip the secondary step of going into the TV control panel. A status

component was also added. The status component gives an overview of the current state of a

device, in this case the DVD Player. It lets the user know that the DVD Player is off, and that

sound is being sent out through TV speakers rather than surround sound.

28

Figure 14: Prototype Two - Revised DVD Player Control Panel

Surround sound (Figure 15) and Alarm Clock (Figure 16) got similar treatment. Surround

sound was made more streamlined by narrowing down the container boxes for all the buttons and

adding a status box. An alarm clock also got a status message box which lets users know whether

the clock is set or not. A “Set” button was also added to the Alarm Clock.

Figure 15: Prototype Two - Revised Surround Sound Control Panel

29

Figure 16: Prototype Two - Revised Alarm Clock Control Panel

Home Security (see Figure 17) again much like the Home Lights section was helped out a

lot by changing the background box color to white which allowed it to contrast more with the

overall application background.

Figure 17: Prototype Two - Revised Home Security Control Panel

That constitutes all the changes made across two prototypes based on user feedback. Next

sections discuss the experimental setup and statistical and qualitative findings.

30

Scoring the System Usability Scale (SUS) 

In addition to retrieving quantitative data such as task success rate and response time to

each task, which can be acquired from the remote usability study, a self-reporting survey on the

overall system usability is useful in determining the overall usability of a system. The use of a

self-reporting survey to gauge overall usability of a system is common practice in the usability

field. Bangor, Kortum & Miller (2008) used a SUS form to test its reliability against other

popular usability form and found that SUS is commonly used amongst usability practitioners.

Although quantitative measures of response time and success rates are critical in determining the

usability of a system, the use of a SUS is an important piece of information that allows us to

glean into users’ perceptions and attitudes.

The SUS is used to produce a single number, which is then used to provide researchers

with a composite score of the system. It is important to note that each score on its own is

meaningless. Sum up all the scores first in order to compute the overall score. Each one will

provide a contribution of anywhere between 0 and 4. Questions 1, 3, 5, 7 and 9 are scored by

taking the score and subtracting 1 from each score. For example, if a participant had marked the

second box, the score would be two for that checkmark. A one is then subtracted from that two

and the overall score of one for this question is yielded. Questions number 2, 4, 6, 8 and 10 are

scored by subtracting the position number from number 5. For example, if a participant had

marked the second box, the score would be two for that checkmark. That two is then subtracted

from 5 and it yields an overall score of 3. The sum of all scores is then multiplied by 2.5 and this

produces an overall system usability score. These scores can only range between 0 and 100. The

easiest way to interpret the SUS score is to use a school grading scale (e.g., 0-60 = fail, 60-70 =

D, 70-80 = C, 80-90=B and 90-100=A). (Bangor, Kortum, & Miller, 2008)

31

The following is the System Usability scale used in this research:

Figure 18: The SUS Survey Used In This Research (In Its Original Form)

32

Literature review indicates availability of other usability scales such as After Scenario

Questionnaire (ASQ) developed by IBM, Computer System Usability (CSUQ) developed by

IBM, Usefulness Satisfaction and Ease of Use (USE) scales (Bangor, Kortum, & Miller, 2008).

These scales are considered effective, however, they often require participants to complete as

many as 50 questions. Most subjective measures up until the SUS form were gained through

attitude scales such as CUSI (Kirakowski and Corbett, 1988). However, the problem with this

scale is that is not specific to any given system because they only measure attitudes.

Because this was an online usability study it was important to know that the participants

were actually reading the questions and understanding what is being asked of them. Due to this

concern, a “speed bump” was introduced into the questionnaire. Speed bumps are commonly

used in testing (Rubin, 1994). It is a question that tests reading comprehension. For example, in

this usability test, as the 8th question participants were asked to “For this question please check 1

(strongly disagree)”. Participants who had read the question would check 1, those who

supposedly skimmed through the entire survey gave this question a different value. Participants

who answered anything but 1 for this question were removed from the study.

Experiment Procedure 

Twenty people participated in the first and second study. Participants were recruited

through social media networks such as Twitter, Facebook and through web sites like

Craigslist.org. The web prototype was developed using Axure RP Pro 5.5 for Macintosh. Axure

was used to develop a fully functional wireframe along with conditional logic built into the

prototype. The wireframe was designed for optimal display at a resolution of 800x600 pixels.

The font used to display the instructions was the standard Arial font.

33

The experiment was automated and conducted remotely by each participant. Loop11

remote usability test application was used to conduct the experiment. The program allows

researchers to create a self-moderated usability study. The service lets researchers set up remote

usability tests by creating their own tasks and pre and post task questions. Each task is displayed

as a frame bar with two buttons “Abandon Task” and “Task Complete”. On the back end,

Loop11 would track mouse clicks, mouse movements and it would measure the response time to

complete each task as well as the path each participant would take to complete the task. After the

participants completed all the tasks they were then forwarded to a qualitative survey. The

qualitative survey used was a System Usability Scale (SUS) a free likert scale developed by

Digital Equipment Corporation. It is a 10-question simple scale used to determine the overall

system usability.

After a study is created, the program generates a link to the online application or web site

that’s being tested. From this point on all data collection and experiment is automated. After

clicking a link to begin the study each participant was given a brief introduction into the study.

This is a study on the ease of use of consumer electronic devices. We will

show you an unfinished prototype and we would like your feedback. This

study is a research study conducted for and on behalf of Northeastern

University in Boston, MA. If you feel you have questions about this

survey or want to learn more about this study please contact Sanel at

selimovic.s@husky.neu.edu or my phone number at xxx-xxx-xxxx. You

will need to set aside 10 to 15 minutes to participate including the post-

study survey. Study consists of 11 tasks. Second part of the study is a

follow-up survey. It should take under 3 minutes to complete.

34

Instructions: Although this application is running on the web, it is

intended to work on a device such as an Apple iPad or a similar tablet

device. You need to use your mouse clicks to indicate your actions.

Because we can’t tell where your mouse is moving you will have to click

on icons and buttons to indicate an action. If you can’t complete the

task, you can use the “Abandon Task” button. Thank you!

Once the participants click the “Begin Evaluation” button, they are given brief

instructions on how to interact with the usability test (Figure 19).

Figure 19: Instructions On How To Use Loop11

35

Task List 

All participants were asked to complete the tasks in Table 2: Table 2: List Of Tasks And Order They Were Presented To The User

For each of the 11 tasks a user will be presented with a task he or she needs to complete.

Once the participant gets to a point in the prototype that they feel they are complete with a given

task they push the “Task Complete” button. In case a participant is not sure what to do or how to

complete a task they are told to push the “Abandon Task” button. Once again, before the task

begins the participants are reminded that they are not being evaluated but that the web site is

being evaluated. Once started, participants go through 11 tasks. Upon completion of 11 tasks

participants are thanked and reminded to take the final survey. The survey consists of a SUS

Task Number

Task Description

Task 1 You want to watch your favorite show on channel 7. How would you do this? Task 4 The neighbors are making a lot of noise and you can’t hear your show. Using the mouse

clicks to indicate the steps, show what you would do. Task 8 You are feeling cold and you would like to make it warmer in the house. Using the

mouse clicks to indicate the steps, show how you would do this. Task 6 You are going to sleep soon and you want the heat to automatically turn off in half an

hour. Using the mouse clicks to indicate the steps, show how you would do this. Task 3 You are already in bed but can’t remember if the lights are on in your kitchen. How do

you make sure the lights are off? Task 5 You don’t want to watch the previews at the beginning of a DVD movie. Using the

mouse clicks to indicate the steps, show what you would do. Task 10 You’ve inserted a new disk into the DVD player, and want to watch a movie. Using the

mouse clicks to indicate the steps, show how you would do this. Task 2 You turned on your video game system but you can’t hear or see anything on the TV.

You know the gaming system is in fully functional. What would you do? Task 7 You need to make sure you wake up by 6:45 a.m. tomorrow. Using the mouse clicks to

indicate the steps, show how you would do this. Task 11 You left for vacation and want to make sure the intrusion alarm is set up. Using the

mouse clicks to indicate the steps, show how you would do this. Task 9 You are at work but you need to let someone into your house. They don’t have your

house key but you need to let them in. Using the mouse clicks to indicate the steps, show how you would do this.

36

scale and a demographic form. Overall study takes on average between 10 to 15 minutes to

complete including the post-task survey. This concludes the experimental procedure.

Chapter 4: Experiment Results

The results of this study appear promising. Based on statistical and qualitative results of

the results, making incremental improvements to design results in an improved overall design.

However, further research is needed to increase user satisfaction with the overall system and to

decrease the overall error rate.

Initial Survey Results 

Before any of the tests had been started, a 27-question survey was administered to

potential users of the system. The first question listed the most frequently found appliances in a

modern home. The participants were asked, “What devices or appliances would you like to be

able to control remotely? Check all that apply.” Participant responses are broken down in Table

3.

37

Table 3: Breakdown Of Participant Responses To Short Survey

  Device Name  Number of Responses  Percentage of Responses 1  TV  16  76% 2  Air Conditioning  15  71% 3  Lights  15  71% 4  DVD Player  14  67% 5  Heater  13  62% 6  Surround Sound  13  62% 7  Alarm Clock  12  57% 8  Home Security System  11  52% 9  Door Locks  10  48% 

10  Stereo  10  48% 11  Clothes Washer  6  29% 12  Dishwasher  6  29% 13  Computer  6  29% 14  Clothes Dryer  5  24% 15  Stove  4  19% 16  Microwave  3  14% 17  Fridge  1  5% 18  Other  1  5% 

Based on the number and percentage of responses, only those devices with one third of

responses or more were selected for implementation in current interface. In total, participants

were given 18 different devices to choose from. In order to develop a prototype low-scoring

devices had to be removed first. The cut-off point for making it into the final prototype was

having one third or more responses. Only 9 devices received a rating of 33% of greater.

Although other devices had received a good number of responses they were not select for

the initial prototype. They were not chosen because upon closer analysis it turned out that even

though participants may have said they want to have remote control of a device, when given a

question detailing the available functionality for that device they did not choose any of the

functionality. For example, for a clothes washer 29% of participants said they wanted remote

38

control of it, however 38% of overall participants listed “None of these” when asked what

functionality they would actually want in a clothes washer.

Majority of participants expressed no interest in remote control of a fridge, saying “Some

things require you to move to the object anyways, like a refrigerator. A remote that brought the

food to the table would be nice.” Other reasons for not wanting remote control of a refrigerator

included human comfort or a need that the object can provide, with one participant saying: “One

of the greatest joys of refrigerators is opening it to see what food is inside, then coming back

minutes later as if expecting to find something new. Having a remote that can check fridge

inventory would destroy that.”

A few participants noted security concerns with the appliances and they felt that things

like a stove were too unsafe to operate remotely “One should never be able to turn on products

that can catch fire, etc. without being in the room. Now turning off only the items may be useful,

but never turning on.”

Due to the limitations of the survey tool it was impossible to segregate individual

participant responses from the group. Based on the response summary from Table 3, the devices

selected for prototyping were as follows: TV, A/C, home lights, heater, surround sound, alarm

clock, door locks and a home security system.

Because surround sound and a stereo system offered similar or replicating features and

functionality, they were grouped into a single device under surround sound. The same decision

was made for home security and locks whose grouping makes logical sense.

Deciding which features to prototype was also contingent on survey responses. Table 4

summarizes the responses for remote TV functionality. Not surprisingly, the most commonly

39

requested feature for a remote controlled TV is the ability to turn the TV on and off, followed by

changing the volume and the audio/video input. All the functionality that was requested by more

than forty percent of respondents was implemented in the final design.

Table 4: Most Frequently Requested TV Features And Functionality

Feature or Functionality Number of Responses

Percent of Responses

Turning the TV on and off 17 81.00% Mute the TV 16 76.00% Changing the volume 16 76.00% Changing the video/audio input 14 67.00% Access the TV menu system to change the TV preferences 14 67.00% Turn the closed captioning on and off 9 43.00% Set a favorite TV channel. 9 43.00% View current TV status 8 38.00% Turning secondary audio programming (SAP) on and off 8 38.00% Setting an automatic timer so that the device can turn on based on/off a schedule

7 33.00%

Other 0 0.00% None of these 0 0.00%

Next set of questions refers to the functionality of a heater and an air conditioner. Most

frequently requested features for the heater and the A/C can be found in Tables 5 and 6,

respectively. All features of both the heater and the A/C were implemented for the first

prototype. Only one participant stated that they do not want to have any remote control over their

heater, while none of the participants had the same thoughts about the A/C. Due to the similarity

in functionality of the two devices, these two devices were combined into a single control panel

in the final prototype design. For both the heater and the A/C majority of respondents, 86% for

each, said they want to be able to adjust the temperature remotely. While 67 and 71% of

participants for heater and the A/C, respectively, wanted the ability to adjust the length of time

the device stays turned on. In other words, participants wanted an auto-off switch.

40

Table 5: Summary Of Responses For The A/C

Feature or Functionality  Number of Responses 

Percent of Responses 

Turn the A/C on or off remotely 18 86% Adjust the A/C temperature 18 86% Adjust the length of time the A/C stays on

15 71%

Adjust the A/C humidifier 13 62% None of these 0 0% Other 0 0%

Table 6: Summary Of Responses For Heater

Feature or Functionality  Number of Responses 

Percent of Responses 

Adjust the heater temperature 18 86% Turn the heater on or off remotely 17 81% Adjust the heater humidifier 14 67% Adjust the length of time the heater stays on

14 67%

None of these 1 5% Other 0 0%

The following table, Table 7, refers to the responses received regarding the lights control

in a home. Although certain lights switches and fixtures come with specialized functionality such

as the ability to dim the lights, the basic functionality of a light bulb has stayed the same since

it’s invention. People either want the lights on or off. Forty-three percent of participants

responded by saying they wanted an ability to check the status of lights in each room and this

functionality was built into the prototype.

41

Table 7: Summary Of Responses For Lights

Feature or Functionality  Number of Responses 

Percent of Responses 

Turn the lights on or off 18 86% Check the status of lights in each room 9 43% None of these 1 5% Other 0 0%

DVD player functionality (Table 8) also received a lot of responses. Like all other

devices, participants requested the ability to turn the DVD player on or off remotely as their top

choice. Next they wanted an ability to play a DVD movie or skip and rewind to next chapter

followed by adjusting the DVD player settings. The DVD player settings were not completely

implemented because of variability between manufacturers and within the brands themselves.

Table 8: Summary Of Responses For DVD Player

Feature or Functionality  Number of Responses 

Percent of Responses 

Turn the DVD player on or off remotely 17 81% Play a DVD movie 16 76% Skip or rewind to the next chapter 15 71% Adjust the DVD player settings 10 48% Have total control of the DVD player remotely without using a DVD remote

8 38%

None of these 1 5% Other 1 5%

Surround sound (Table 9) and a stereo system were combined into a single control panel.

Results for the surround sound only are displayed below. Majority of participants requested an

ability to change the volume and the audio/video inputs on the two devices, while a substantial

number of participants also said they would like to view current operational status of the

surround sound.

42

Table 9: Summary Of Responses For Surround Sound

Feature or Functionality  Number of Responses 

Percent of Responses 

Turn the surround sound on or off 15 71% Adjust the volume 14 67% Change the video/audio input of surround sound

13 62%

View status of the surround sound 10 48% None of these 2 10% Other 0 0%

Home security (Table 10) and locks (Table 11) were originally proposed as two separate

devices and two different control panels. However, upon examining the functionality offered by

both devices, it made logical sense to group the functionality of the two devices into one. Home

security systems and locks are often tied together in people’s minds and putting locks together as

part of security seems to have made sense.

Table 10: Summary Of Responses For Home Security

Feature or Functionality  Number of Responses 

Percent of Responses 

Turn the alarm on or off remotely 13 62% Set the alarm remotely 11 52% View alarm settings remotely without being able to change them

8 38%

None of these 5 24% Other 1 5% Table 11: Summary Of Responses For Door Locks

Feature or Functionality  Number of Responses 

Percent of Responses 

View whether the house is locked or unlocked

16 76%

View overall status of home security 16 76% Arm or disarm the home security system remotely

13 62%

Lock or unlock the home remotely 11 52% None of these 3 14% Other 0 0%

43

System Usability Scale (SUS) Results

The experimental results of the post study questionnaire are very positive. All twenty

participants (ten for prototype one and ten for prototype two) completed the System Usability

Scale (SUS) survey and the demographic questionnaire. The questions were presented exactly as

they are shown in the SUS survey (see Figure 18) with the addition of a speed bump question

whose purpose is to detect any users who might simply be skimming through the form and not

reading the questions themselves. It’s important to note that these results do not serve as the only

factor in determining the usability of a system or application. The SUS should be used in

conjunction with other performance data such as average response time and number of errors. On

it’s own, however, the SUS score offers a good indication of the overall usability and any trends.

SUS Results Analysis 

First, the answers to the questionnaires were rated. The weight of each question was

determined according to the instructions provided by Brooke (1996). For example, questions

number 1, 3, 5, 7 and 9 are scored by taking the value a participant had marked, which could be

anywhere from 1 to 5 and subtracting 1 from each score. Questions number 2, 4, 6, 8 and 10 are

scored by subtracting the value a participant had marked from a total number of 5. If a

participant had answered “Somewhat Disagree” which has a value of two, then a two would be

subtracted from number five giving a total value of 3 to that answer. The scores for each question

were then tabulated and overall score calculated based on that table.

Table 12 contains the SUS scores for each of the ten participants across all ten questions

for Prototype One only. The participants had to choose between five options on a scale ranging

from “Strongly Disagree” (1) to “Strongly Agree” (5).

44

Table 12: Prototype One - Summary Of Responses To The SUS Survey

Participant q1 q2 q3 q4 q5 q6 q7 q8 q9 q10 SUS Score

1 4 2 4 1 4 1 4 2 4 2 80.0 2 4 3 3 2 3 2 4 2 3 2 65.0 3 3 3 4 1 4 2 3 3 4 4 62.5 4 5 1 5 1 5 1 5 5 5 1 90.0 5 2 1 5 1 5 1 5 1 5 4 85.0 6 3 1 4 1 5 1 5 1 5 1 92.5 7 3 2 4 3 4 4 4 2 2 3 57.5 8 3 2 4 2 3 1 4 2 4 2 72.5 9 2 4 3 1 2 2 4 1 4 4 57.5 10 3 4 2 2 3 3 3 3 3 4 45.0 Overall SUS 70.8

The last column labeled “SUS Score” is an excel formula which takes all the values in a

row and calculates the SUS Score and displays it as a percentage for each of the ten participants

for Prototype One. The last row and column displays an overall SUS score. The SUS Score for

Prototype One was 70.8%, which according to Figure 2 falls in between marginal and good

categories. The lowest SUS score belongs to participant 10 at 45% overall and the highest

recorded SUS score was 92.5% for participant 6. Statistical data are further summarized in Table

13.

Table 13: Prototype One - Summary Of Statistical Data For SUS Survey

Mean SD Median Mode Group 1 Participants 70.8 16.4 68.75 57.5 Experiment two survey was exactly the same as experiment one. The second experiment

consisted of ten participants who were recruited online and at the end of the eleven tasks they

were asked to complete the SUS. Data collected from Prototype Two are summarized in Table

14 below. Table 14 contains the SUS scores for each of the ten participants across all ten

questions for Prototype Two only. Just as in Prototype One, participants were asked to rate the

usability of the system on a scale of (1) “Strongly Disagree” to (2) “Strongly Agree.”

45

Table 14: Prototype Two - Summary Of Responses To The SUS Survey

It is evident from comparing the two tables that data for Prototype Two result in much

larger individual SUS Scores and a greater overall SUS Score. The total overall SUS Score for

Prototype Two is 79.8%. For all the participants in the first experiment, the mean SUS score was

70.8%, which is a total overall increase from Prototype One by 9%. The last row and column

gives an overall SUS score. The lowest SUS score belongs to participant 3 at 57.5 percent overall

and the highest recorded SUS score was 100 for participants 2 and 7. Statistical data are further

summarized in Table 15.

Table 15: Prototype Two - Summary Of Statistical Data For SUS Survey

Mean SD Group 2 Participants 79.8 17.8

For all the participants in the first experiment, the mean SUS score was 70.8 with a

standard deviation of 16.4. The results of the second study based on the feedback and

improvements done to the original design indicate a substantial increase in the overall System

Usability Survey score. A passing mark for any given system is 70% and 79.8% is just .2% shy

of 80% needed to fall into the “excellent” category.

Participant q1 q2 q3 q4 q5 q6 q7 q8 q9 q10 SUS Score 1 4 1 5 1 4 1 5 1 5 1 95.0 2 5 1 5 1 5 1 5 1 5 1 100.0 3 4 3 4 3 3 4 3 2 4 3 57.5 4 2 2 4 1 3 2 4 2 4 1 72.5 5 3 1 4 2 4 2 4 3 4 2 72.5 6 4 1 4 1 4 1 4 1 4 1 87.5 7 5 1 5 1 5 1 5 1 5 1 100.0 8 3 3 4 1 3 2 4 2 4 2 70.0 9 2 3 3 1 2 3 4 5 3 2 50.0 10 4 1 4 1 5 1 5 1 4 1 92.5 Overall SUS 79.8

46

Performance Data Results 

General Results  

Performance data refers to the data collected by the remote usability application. This

includes participant’s response times (e.g., how long it took them to complete each task) and to

participant’s error rates (e.g., how many participants completed the tasks correctly). These data

allow greater insight into how the tests went, such as what tasks produced greatest error rates and

which tasks had the longest or shortest completion times. Average task completion times will be

considered first. Table 16 provides an overall view of average completion times in seconds.

Table 16: Average Task Completion Time

Completion Time in Seconds   Test 1 Test 2 Task 1 51.3 46.30 Task 2 85.1 64.30 Task 3 39.9 27.00 Task 4 54.7 37.70 Task 5 36.7 37.40 Task 6 50.2 31.80 Task 7 40.2 31.50 Task 8 38.0 22.90 Task 9 51.0 26.40 Task 10 29.9 19.00 Task 11 42.1 26.70

This same data is summarized in Figure 20. It is evident from the figure that almost every

task in experiment 2 was completed in less time than the same task in experiment 1. The taller

the bar, the more time it took to complete a task, therefore, shorter is better in this instance. For

Task 5, however, it appears that the completion time was slightly longer than in Experiment 1. In

Experiment 1 task five took 36.7 seconds to complete and in Experiment 2 it took 37.40 seconds

to complete. This could be attributed to the fact that some participants expressed difficulty with

47

completing this task, and although both groups managed to complete the task, the amount of time

it took them to understand the question and find the information was the same.

Figure 20: Average Task Completion Time In Seconds Across Tasks

Task success rate can be viewed in two ways. The first way to look at it is as a percent of

tasks each participant completed successfully. The second way to look at it is as a percent of

participants who completed each task successfully. The percent of tasks each participant

completed successfully can be found in the right column of Table 17. The percent of participants

who completed each task successfully can be found in the last row of Table 17. The remote

usability application can’t provide an automated error table because of the limitations in

technology. Error rates for each participant were derived through participant session video

recordings available through the application.

0 10 20 30 40 50 60 70 80 90 

1  2  3  4  5  6  7  8  9  10  11 

Second

s 

Task 

Average Task CompleXon Time (in seconds) 

Experiment 1 

Experiment 2 

48

Table 17: Average Number Of Errors Across Participants And Across Tasks For Experiment 1

Participant  Task 1  Task 2  Task 3  Task 4  Task 5  Task 6  Task 7  Task 8  Task 9  Task 10  Task 11  Average 

P1 0  1  0  0  1  1  1  1  1  1  1  73% 

P2 1  0  1  1  1  0  1  0  0  1  1  64% 

P3 1  0  1  1  0  1  1  1  0  1  1  73% 

P4 1  1  1  1  1  1  1  1  1  1  1  100% 

P5 1  1  0  0  1  1  1  1  1  0  1  73% 

P6 1  0  1  1  0  1  1  0  1  1  0  64% 

P7 1  1  1  1  1  1  1  1  1  1  1  100% 

P8 1  0  1  0  0  1  1  1  1  1  1  82% 

P9 1  1  1  1  1  1  1  1  1  1  1  100% 

P10 0  1  0  1  1  1  1  1  1  1  0  73% 

Average 80%  60%  70%  70%  70%  90%  100%  80%  80%  90%  80%   

Table 17 and Table 18 are binary tables. A one “1” denotes a successful task completion

and a zero “0” denotes task failure. The total number represents the total number participants

who successfully completed a given task. A total of 8 would then mean that a total of 8 out of 10

participants successfully completed this task.

The lowest-scoring task across all participants was task two “You turned on your video

game system but you can’t hear or see anything on the TV. You know the gaming system is in

fully functional. What would you do?” The can be attributed to not understanding the question

well. Many participants went for the TV functionality and attempted to fix it there. The correct

path would have been to go to surround sound and select a “Gaming” device as the input for the

surround sound, however, instructions did not make it clear that the gaming system and the DVD

player are both connected through the Surround Sound. All participants successfully completed

task seven: “You need to make sure you wake up by 6:45 a.m. tomorrow. Using the mouse clicks

to indicate the steps, show how you would do this.” This was the simplest task to complete out of

all eleven tasks because the path was clear, the participants had to go to the alarm clock panel

49

and it was clear what to do there because of the limited features and functionality of the alarm

clock.

The last column represents the total number of errors produced by each participant across

all tasks. Again these data are binary and one “1” denotes a successful task completion and zero

“0” denotes a task failure. The total number in the right column represents the total number of

successfully completed tasks out of 11. A total of 9 would then mean that a total of 9 out of 11

participants successfully completed this task. The lowest-performing participant between all ten

was participant number two with only 64% of correct answers.

Based on the feedback from experiment one or prototype one, a second prototype was

created and retested with the same test. Second prototype attempts to address shortcomings of

the first prototype. Its primarily goal was to create a more streamlined task scenarios for each of

the components. As a result various “Steps” were added to each of the components and

subcomponents of the main menu were grouped logically so that subcomponents such as “Turn

TV Off” and “On” were pulled out and grouped into step one. This panel in particular was taken

out and given it’s own step because many participants would change the settings for a given

device and turn on the device or appliance after making changes to all the settings. Based on

these observations a second prototype was created and a second experiment was conducted.

The following table (18) presents all the success rates for ten participants in experiment

two or prototype two. In some ways data are similar to experiment one. For example, task two:

“You turned on your video game system but you can’t hear or see anything on the TV. You know

the gaming system is in fully functional. What would you do?” was the lowest scoring task

among all the tasks.

50

Table 18: Average Number Of Errors Across Participants And Across Tasks For Experiment 2

Participant  Task 1  Task 2  Task 3  Task 4  Task 5  Task 6  Task 7  Task 8  Task 9  Task 10  Task 11  Average 

P1  1  1  1  1  1  1  1  1  1  1  1  100% 

P2  1  1  1  1  1  1  1  1  1  1  1  100% 

P3  1  0  1  0  1  1  1  1  1  1  1  82% 

P4  1  0  1  1  1  1  0  1  1  1  1  82% 

P5  1  1  1  1  1  1  1  1  1  1  1  100% 

P6  1  1  1  1  1  1  1  1  1  1  1  100% 

P7  1  1  1  1  0  1  1  1  1  1  1  91% 

P8  1  0  1  1  1  1  1  1  1  1  1  91% 

P9  1  0  1  1  0  1  1  1  1  1  1  82% 

P10  1  1  1  1  1  1  1  1  1  1  1  100% 

Average 100%  60%  100%  90%  80%  100%  90%  100%  100%  100%  100%   

Figures 21 and 22 offer another view of success rates across tasks and across participants. In

almost all situations, participants completed a greater number of tasks in prototype two than in

one. Figure 22 offers a view across participants and although three participants did better in the

first prototype than in second, overall success rate was greater for prototype two.

Figure 21: Successful Task Completion Across Participants

0 

2 

4 

6 

8 

10 

12 

1  2  3  4  5  6  7  8  9  10  11 

Num

ber o

f ParXcipan

ts 

Task Number 

Success Rate Across Tasks between two prototypes 

Prototype 1 

Prototype 2 

51

Figure 22: Successful Task Completion Across Participants

Table 19: Experiment 1 – Mean Task Ease Of Use Ratings

  Task 1  Task 2  Task 3  Task 4  Task 5  Task 6  Task 7  Task 8  Task 9  Task 10  Task 11  Mean 

p4  2  2  3  2  2  2  2  2  2  2  2  2.09 p5  3  2  2  2  3  1  2  2  1  2  2  2.00 p6  2  0  2  2  1  2  2  2  2  2  2  1.73 p7  3  2  3  1  2  3  3  3  3  3  3  2.64 p8  3  2  3  2  2  3  3  3  3  3  3  2.73 p10  2  2  2  2  2  2  2  2  2  3  2  2.09 p12  3  2  2  2  0  3  3  3  2  2  2  2.18 p13  2  0  3  2  1  1  3  2  2  2  2  1.82 p15  2  1  2  2  0  1  2  2  1  0  1  1.27 p16  0  1  3  2  2  3  3  3  3  3  2  2.27 Mean  2.2  1.4  2.5  1.9  1.5  2.1  2.5  2.4  2.1  2.2  2.1   90% CL  0.48  0.44  0.27  0.16  0.51  0.46  0.27  0.27  0.38  0.48  0.30   

Table 20 tabulates data on participants task ease of rating along with means across tasks and

across participants for Experiment 2.

0 

2 

4 

6 

8 

10 

12 

1  2  3  4  5  6  7  8  9  10 

Task Com

pleX

on Rate 

ParXcipant Number 

Success Rate Across ParXcipants between  two prototypes 

Prototype 1 

Prototype 2 

52

Table 20: Experiment 2 – Mean Task Ease Of Use Ratings

  Task 1  Task 2  Task 3  Task 4  Task 5  Task 6  Task 7  Task 8  Task 9  Task 10  Task 11  Means 

p1  3  2  3  2  2  2  3  3  3  3  2  2.5 p2  3  2  3  2  3  3  3  3  3  3  3  2.8 p3  2  1  2  2  1  2  3  3  2  2  2  1.9 p4  3  1  3  1  2  1  2  3  2  2  3  2.1 p5  2  2  3  2  2  3  2  2  2  3  3  2.4 p6  2  3  2  2  2  2  2  2  2  2  2  2.0 p7  3  1  3  2  0  3  3  3  3  3  3  2.5 p8  2  1  3  2  1  2  2  3  3  2  3  2.1 p9  2  1  2  2  0  2  2  2  3  2  2  1.8 

p10  3  2  3  2  2  3  3  1  3  3  3  2.5 Means  2.5  1.6  2.7  1.9  1.5  2.3  2.5  2.5  2.6  2.5  2.6   90% ci  0.27  0.43  0.25  0.16  0.51  0.35  0.27  0.37  0.27  0.27  0.27   

Figure 24 illustrates the differences between the two prototypes on the mean ease of use task

ratings.

Figure 23: Mean Ease Of Task Rating, Higher Values Are Better

0 

0.5 

1 

1.5 

2 

2.5 

3 

Task 1  Task 2  Task 3  Task 4  Task 5  Task 6  Task 7  Task 8  Task 9  Task 10 Task 11 

Mean Ease Task RaXng (0‐3, Higher = Be^er) 

Prototype 2 

Prototype 1 

53

Statistical Analysis 

It is important to cover statistical analysis of all tasks and ratings. T-test statistic was used

to analyze the number of errors across tasks, participants and average task completion times. T-

test is used instead of an Analysis of Variance (ANOVA) because ANOVA should only be used

when testing 3 or more groups. These experiments revolve around comparing two different design

prototypes and therefore, t-test was deemed the most appropriate.

Overall, all differences were found to be statistically significant. This includes the task

completion time in seconds (P = 0.0323) (Table 22), number of errors across tasks (P = .0206 )

(Table 24) and number of errors across participants (P = .0287) (Table 26)

All t-tests completed are two-tailed tests. The first t-test was done to compare the average

time it took each participant to complete all the tasks. Data on completion time was collected

through Loop11. A task begins once a page is loaded and ends when a participant pushes the

“Task Complete” button. The numbers in Table 19 are a cumulative average of all ten participants

across both prototypes.

Table 21: Average Number Of Seconds To Complete A Task

Prototype 1 Prototype 2 Task 1 51.3 46.30 Task 2 85.1 64.30 Task 3 39.9 27.00 Task 4 54.7 37.70 Task 5 36.7 37.40 Task 6 50.2 31.80 Task 7 40.2 31.50 Task 8 38.0 22.90 Task 9 51.0 26.40 Task 10 29.9 19.00 Task 11 42.1 26.70

54

Table 20 offers a statistical summary of all the data gained from the table above. The

significance level of an unpaired t-test for data is less than the necessary p < .05 level. The

differences in task completion time are, in fact, significant at p = 0.0323 level, far below the level

necessary to constitute statistical significance. This indicates that the two mean values are indeed

truly different from one another.

Table 22: Unpaired T-Test Of Average Completion Time In Seconds

Group  Prototype 1  Prototype 2 Mean 47.1909 33.7273 SD 14.681 12.701 N 11 11 P-value P = 0.0323

The second t-test was done to compare the number of participants who managed to

successfully complete each task between the two prototypes. A higher number in this instance is

better because it means a greater number of participants completed a given task. Ten is a

maximum number. As noted in the earlier analysis data on successful completion of each task

was collected by observing video playback of each participant’s study and denoting his or her

success with a 1 or failure with a 0. The numbers in Table 23 represent a total number of

participants (out of ten) who completed each task on each of the two runs.

Table 23: Success Rate Across Tasks

  Prototype 1 Prototype 2 Task 1 8 10 Task 2 6 6 Task 3 7 10 Task 4 7 9 Task 5 8 8 Task 6 9 10 Task 7 10 9 Task 8 8 10 Task 9 8 10 Task 10 9 10 Task 11 8 10

55

Table 24 offers a statistical summary of all the data gained from the table above. The

significance level of an unpaired t-test for data in Table 22 is less than the necessary p < .05

level. The differences in task completion time are, in fact, significant at p = 0.0206 level, far

below the level necessary to constitute statistical significance. This indicates that the two mean

values are indeed truly different from one another.

Table 24: Unpaired T-Test Of Success Rate Across Tasks

  Group  Prototype 1    Prototype 2   Mean 8.00 9.27 SD 1.10 1.27 N 11 11 P-value P = .0206

Table 25: Number Of Errors Across Participants

Table 26: Unpaired T-Test Of Task Completion Across Participants

  Group    Prototype 1      Prototype 2   Mean 8.80 10.20 SD 1.62 0.92 N 10 10 P-value P = .0287

Combined, these results are very positive. The significance level of each of these tests

means that there were valuable and valid improvements made to prototype one which resulted in

Prototype 1 Prototype 2

p1 8 11

p2 7 11

p3 8 9

p4 11 9

p5 8 11

p6 7 11

p7 11 10

p8 9 10

p9 11 9

p10 8 11

56

a better overall user experience in prototype two or experiment two. Research indicates that

participants in experiment two outperformed participants in experiment one in almost all cases.

The significance level for this finding is p = .0323. With regards to task success rates, these two

measures were also statistically significant. A comparison of tasks completed successfully by

each participant yields a statistically significant results of p = .0206. A comparison of

participants who completed each task successfully yields a statistically significant result of p =

.0287.

Overall Summary 

As documented in the discussion, Tables 22, 24, and 26, summarize the results of the

study. Based on the statistical analysis of differences between the two groups the results of the

study are significant and point to an overall improvement from the original design. There are

some unknowns that warrant further study and examination. For example, participants who

tested prototype one did better than participants in prototype two for task five. This difference

however is only several milliseconds so it could likely be ascribed to random error.

Chapter 5: Conclusion 

The number of tablet devices such as iPads sold worldwide has grown to 4.19 (Apple Inc,

2010) million and with the introduction of Android platform on tablet devices the potential

market opportunity for a device that combines total home control in one interface is even greater.

The ability of these devices to remain friendly to abled and disabled consumers is critical.

This research discussed two different incremental design improvements, error rates, task

completion time and an overall System Usability Survey score. Across all measures the results

yielded statistically significant findings. An almost ten point swing in the overall System

57

Usability Scale (SUS) survey puts prototype two on the border of an excellent category. This

means that with further adjustments and testing a prototype could be made to match user’s

expectations much better. With respect to the overall results on mean completion time, statistical

analysis indicates that these results are indeed significant at the p<.05 level, and the same holds

true for statistical comparison across tasks and across participants.

This study is a move in positive direction in opening up the discussion about what

common functionality all devices should share and what is the best method for portraying that

functionality to users. The initial survey requesting participant responses did well in gauging user

interest in different appliance functionality. The survey helped identify some critical features that

users want from their devices and shed light on some features which are generally considered to

be wanted by users but the users never expressed any want or need for them. In a future study the

list of tasks and scenarios should be further considered before testing and implementation. A

future study should consist of a more statistically representative sample of general population.

There are many benefits to running a between subjects study chief of which is easier recruitment

pool for a second follow-up study, however it would be interesting to re-run the study as a within

subjects design rather than a between subjects design. Although an online or remote usability

study has proved useful in this situation, more valuable insight could be gleaned from live in-

person usability studies.

58

Appendix 

Appendix A: SUS Survey 

59

Appendix B: Instructions to Participants 

This is a study on the ease of use of consumer electronic devices. We will show you an

unfinished prototype and we would like your feedback. This study is a research study conducted

for and on behalf of Northeastern University in Boston, MA. If you feel you have questions

about this survey or want to learn more about this study please contact Sanel at

selimovic.s@husky.neu.edu or my phone number at xxx-xxx-xxxx. You will need to set aside 10

to 15 minutes to participate including the post-study survey. Study consists of 11 tasks. Second

part of the study is a follow-up survey. It should take under 3 minutes to complete.

60

References 

1. Alexandersson, J., Zimmermann, G., and Bund, J. (2009). User Interfaces for AAL: How

Can I Satisfy All Users? In Proceedings of Ambient Assisted Living - AAL. 2. Deutscher

Kongress mit Ausstellung/ Technologien - Anwendungen - Management, pp. 5-9.

2. Apple Inc (October 18, 2010). “Record Mac, iPhone and iPad Sales Highest Revenue and

Earnings Ever.” Press Release. Retrieved November 18, 2010.

http://www.apple.com/pr/library/2010/10/18results.html.

3. Albert, Tullis, Tedesco (2010) Beyond the Usability Lab, p. 131.

4. Backgrounder on the Three Mile Island Accident. (2009). http://www.nrc.gov/reading-

rm/doc-collections/fact-sheets/3mile-isle.html#sources.

5. Bangor, A., Kortum, P. and Miller, J.A. (2008) The System Usability Scale (SUS): An

Empirical Evaluation. International Journal of Human-Computer Interaction, pp. 114-

123.

6. Blum, A., Furst, M. (1997). Fast Planning Through Planning Graph Analysis. Proc. Of

the 14th International Joint Conference on Artificial Intelligence (IJCAI 1997), Montreal,

Canada, pp. 1636-1642.

7. Bolt, N., & Tulatimutte, T. (2010). Remote Research: Real users, real time, real research,

p. 35.

8. Brooke, J. (1996). SUS: A “quick and dirty” usability scale. In P. W. Jordan, B. Thomas,

B. A.

61

9. Davidoff, S., Lee, M.K., Yiu, C., Zimmernan, J., & Dey, A.K. (2006) Principles of smart

home control, to appear in Proceedings of Ubicomp, 2006.

10. Geeksquad, http://www.geeksquad.com.

11. Gershenfeld, N. (2000). When things start to think. Owl Books.

12. Gellersen, H. W., Schmidt, A., & Beigl, M. (2000). Adding some smartness to devices

and everyday thngs. In Proceedings of the third workshop on mobile computing system

and applications.

13. Grinter, R.E., Ducheneaut, N., Edwards, W.K., Newman, M. (2005). The Work To Make

The Home Network Work. In: Proc. Ninth European Conference on Computer-Supported

Cooperative Work (ECSCW 05), pp. 469–488.

14. Hopper, A. (1999). Sentient computing. The royal society Clifford Paterson lecture.

15. Kiesler, S., Lundmark, V., Zdaniuk, B., Kraut, R.E. (2000). Troubles with the Internet:

The dynamics of help at home. Human Computer Interaction, pp. 323–351.

16. Kindberg, T. (2000). People, places, things: Web presence for the real world. In

Proceedings of the third workshop on mobile computing system and applications.

17. Kirakowsi, J, and Corbett, M. (1988), Measuring User Satisfaction, in D M Jones and R

Winder (Eds.) People and Computers IV. Cambridge: Cambridge University Press.

18. Lasky, M. S., McCracken, H., and Bielski, V. (1998). Just how helpful are handhelds?

CNN Interact.

19. Laszlo, J., et al. (2002).V. Home Networking: Seizing Near-Term Opportunities to

Extend Connectivity to Every Room. Jupiter Research.

62

20. Liau Yun Qing. (2010). http://www.zdnetasia.com/enterprises-face-device-

fragmentation-challenge-62204766.htm.

21. Lieberman, H., Liu, H., Singh, P., and Barry, B. (2005). Beating Common Sense into

interactive Applications. AI Magazine 25(4): Winter 2005, pp. 63-76.

22. Lieberman, H., and Espinosa, J. (2006). A goal-oriented interface to consumer electronics

using planning and commonsense reasoning. In IUI ’06: Proceedings of the 11th

international conference on Intelligent user interfaces, pp. 226–233, New York, NY,

USA. ACM Press.

23. Meshkati, N. (1991). Human factors in large-scale technological systems' accidents:

Three Miles Island, Bhopal, and Chernobyl. Industrial Crisis Quarterly, pp. 133-154.

24. Myers, B., Hudson S., and Pausch, R. (2000). Past, present, and future of user interface

software tools. ACM Trans. Computer-Human Interaction, pp. 3-28.

25. Nakajima, T., Ishikawa, H., & Tokunaga, E. (2002). Technical Challenges for Building

Internet-Scale Ubiquitous Computing. In Proceedings of International Workshop on

Object-Oriented Real-Time Dependable Systems.

26. Norman, D. (2007). The design of future things. Basic Books. pp. 152, 23.

27. Pier, K. and Landay, J. (1992). Issues for location-independent interfaces. Tech. Rep.

ISTL92-4. Xerox PARC, Palo Alto, CA. Available at

http://www.cs.berkeley.edu/˜landay/research/publications/LII.ps.

28. Raimer, J. (2005). A history of the GUI. Arstechnica.com. Retrieved on November 30,

2010.

63

29. Raskin, J. (2000). The Humane Interface: New Directions for Designing Interactive

Systems. Addison-Wesley Professional, First Edition, 2000.

30. Reed, P., Holdaway, K., Isensee, S., Buie, E., Fox, J., Williams, J., Lund, A. (1999). User

Interface Guidelines and Standards: Progress, Issues, and Prospects. Interacting with

Computers, pp. 119–142.

31. Rich, C., Sidner, C., Lesh, N., Garland, A., Booth, S., and Chimani, M. (2005).

DiamondHelp: A Graphical User Interface Framework for Human-Computer

Collaboration, in Proc. of IEEE International Conference on Distributed Computing

Systems Workshops, pp. 514-519.

32. Rich, C. (2009). Building task-based user interfaces with ANSI/CEA-2018. Computer,

pp. 20–27.

33. Rich, C., & Sidner, C. (1998). Collagen: A Collaboration Manager Software Interface

Agents. Journal of User Modeling and User-Adapted Interaction, pp. 315-350.

34. Rubin, J. (1994). Handbook of Usability Testing: How to plan design and conduct

effective tests. Wiley, p. 124.

35. Scherf, K. (2002). Consumer Electronics Association (CEA) Conference, Parks

Associates Panel on Home Networking.

36. Shehan, E., Edwards, W.K. (2007). Home Networking and HCI: What Hath God

Wrought? In: Proceedings of Conference on Human Factors in Computing Systems, pp.

547–556.

64

37. Singh, P. (2002). The public acquisition of Commonsense knowledge. Proc. AAAI

Spring Symposium on Acquiring (and Using) Linguistic (and World) Knowledge for

Information Access.

38. Shneiderman, B., and Plaisant, C. (2005). Designing the User Interface: Strategies for

Effective Human-Computer Interaction. Addison-Wesley, Reading, MA.

39. Wang, H. (2006). Networks in the Home: Connected Consumer Electronics. Parks

Associates.

40. Want, R., Schilit, B., Adams, N., Gold, R., Petersen, K., Ellis, J., Goldberg, D., &

Weiser, M. (1995). The ParcTab ubiquitous computing experiment. Technical report

CSL-95-1, Xerox Palo Alto Research Center.