Evaluation

11-19-2012

www.id-book.com

Controlled settings involving users, eg usability testing & experiments in laboratories and living labs.

Natural settings involving users, eg field studies to see how the product is used in the real world.

Any settings not involving users, eg consultants critique; to predict, analyze & model aspects of the interface analytics.

While informal demos to colleagues or customers can provide some useful feedback, more formal expert reviews have proven to be effective

Expert reviews entail one-half day to one week effort, although a lengthy training period may sometimes be required to explain the task domain or operational procedures

There are a variety of expert review methods to choose from: Heuristic evaluation Guidelines review Consistency inspection Cognitive walkthrough Metaphors of human thinking Formal usability inspection

Expert reviews can be scheduled at several points in the development process when experts are available and when the design team is ready for feedback.

Different experts tend to find different problems in an interface, so 3-5 expert reviewers can be highly productive, as can complementary usability testing.

The dangers with expert reviews are that the experts may not have an adequate understanding of the task domain or user communities.

Even experienced expert reviewers have great difficulty knowing how typical users, especially first-time users will really behave.

The emergence of usability testing and laboratories since the early 1980s

Usability testing not only sped up many projects but that it produced dramatic cost savings.

The movement towards usability testing stimulated the construction of usability laboratories.

A typical modest usability lab would have two 10 by 10 foot areas, one for the participants to do their work and another, separated by a half-silvered mirror, for the testers and observers

Participants should be chosen to represent the intended user communities, with attention to background in computing, experience with the task, motivation,

education, and ability with the natural language used in the interface.

Participation should always be voluntary, and informed consent should be obtained.

Professional practice is to ask all subjects to read and sign a statement like this one: I have freely volunteered to participate in this experiment. I have been informed in advance what my task(s) will be and

what procedures will be followed. I have been given the opportunity to ask questions, and

have had my questions answered to my satisfaction. I am aware that I have the right to withdraw consent and to

discontinue participation at any time, without prejudice to my future treatment.

My signature below may be taken as affirmation of all the above statements; it was given prior to my participation in this study.

Institutional Review Boards (IRB) often governs human subject test process

Videotaping participants performing tasks is often valuable for later review and for showing designers or managers the problems that users encounter. ◦ Use caution in order to not interfere with participants ◦ Invite users to think aloud (sometimes referred to as

concurrent think aloud) about what they are doing as they are performing the task.

Many variant forms of usability testing have been tried:

◦ Paper mockups ◦ Discount usability testing ◦ Competitive usability testing ◦ Universal usability testing ◦ Field test and portable labs ◦ Remote usability testing ◦ Can-you-break-this tests

In this eye-tracking setup, the participant wears a helmet that monitors

and records where on the screen the participant is looking

More portable eye-tracking devices

Written user surveys are a familiar, inexpensive and generally acceptable companion for usability tests and expert reviews.

Keys to successful surveys Clear goals in advance Development of focused items that help attain the

goals.

Users could be asked for their subjective impressions about specific aspects of the interface such as the representation of: task domain objects and actions syntax of inputs and design of displays.

Other goals would be to ascertain

◦ users background (age, gender, origins, education, income)

◦ experience with computers (specific applications or software packages, length of time, depth of knowledge)

◦ job responsibilities (decision-making influence, managerial roles, motivation)

◦ personality style (introvert vs. extrovert, risk taking vs. risk aversive, early vs. late adopter, systematic vs. opportunistic)

◦ reasons for not using an interface (inadequate services, too complex, too slow)

◦ familiarity with features (printing, macros, shortcuts, tutorials)

◦ their feeling state after using an interface (confused vs. clear, frustrated vs. in-control, bored vs. excited).

Scientific and engineering progress is often stimulated by improved techniques for precise measurement.

Rapid progress in the designs of interfaces will be stimulated as researchers and practitioners evolve suitable human-performance measures and techniques.

Controlled experiments can help fine tuning the human-computer interface of actively used systems.

Performance could be compared with the control group.

Dependent measures could include performance times, user-subjective satisfaction, error rates, and user retention over time.

www.id-book.com

Physiological measures were used.

Players were more engaged when playing against another person than when playing against a computer.

What precautionary measures did the evaluators take?

www.id-book.com

high values indicate more variation

Playing against

computer

Playing against

friend

Mean St. Dev. Mean St. Dev.

Boring 2.3 0.949 1.7 0.949

Challenging 3.6 1.08 3.9 0.994

Easy 2.7 0.823 2.5 0.850

Engaging 3.8 0.422 4.3 0.675

Exciting 3.5 0.527 4.1 0.568

Frustrating 2.8 1.14 2.5 0.850

Fun 3.9 0.738 4.6 0.699

Source: Mandryk and Inkpen (2004).

• Scenarios ― an informal narrative story, simple, ‘natural’,

personal, not generalisable

• Use cases — assume interaction with a system — assume detailed understanding of the

interaction

• Essential use cases — abstract away from the details — does not have the same assumptions as use

cases

Task descriptions are often used to envision new systems or devices

Task analysis is used mainly to investigate an existing situation

It is important not to focus on superficial activities What are people trying to achieve? Why are they trying to achieve it? How are they going about it?

Many techniques, the most popular is Hierarchical Task Analysis (HTA)

Involves breaking a task down into subtasks, then sub-sub-tasks and so on. These are grouped as plans which specify how the tasks might be performed in practice

HTA focuses on physical and observable actions, and includes looking at actions not related to software or an interaction device

Start with a user goal which is examined and the main tasks for achieving it are identified

Tasks are sub-divided into sub-tasks

0. In order to borrow a book from the library

1. go to the library

2. find the required book

2.1 access library catalogue

2.2 access the search screen

2.3 enter search criteria

2.4 identify required book

2.5 note location

3. go to correct shelf and retrieve book

4. take book to checkout counter

plan 0: do 1-3-4. If book isn’t on the shelf expected, do 2-3-4.

plan 2: do 2.1-2.4-2.5. If book not identified do 2.2-2.3-2.4.

Borrow a

book from

the library

go to the

library

find required

book

retrieve

book from

shelf

take book

to counter 3 2 1 4

0

access

catalog

access

search

screen

enter

search

criteria

identify

required

book

note

location

plan 0: do 1-3-4. If book isn’t on the shelf expected, do 2-3-4.

plan 2: do 2.1-2.4-2.5. If book not identified from information available, do 2.2-2.3-2.4-2.5

2.1 2.2 2.3 2.4 2.5

■ Scenarios, use cases and essential use cases can be used to articulate existing and envisioned work practices.

■ Storyboards can be generated from scenarios

■ Card-based prototypes can be generated from use cases

■ Task analysis techniques such as HTA help to investigate existing systems and practices