Usability Testing

Usability Testing

• What is Usability?• What is Usability Testing?• Design the Test• Prepare for the Test• Perform the Test• Process the Data

What is Usability?• Usability is the measure of the quality of a

user’s experience when interacting with a product or system.

• Usability is a quality attribute that assesses how easy user interfaces are to use.

• The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

What is Usability?• Ease of learning—How fast can a user learn to

accomplish basic tasks?

• Efficiency of use—How fast can an experienced user accomplish tasks?

• Memorability—Can a user remember enough to use it effectively the next time?

• Error frequency and severity—How often do users make errors, how serious are these errors, and how do users recover from these errors?

• Subjective satisfaction—How much does the user like using the system?

What is Usability?

• Quality of experience– Excellence of user experience– Nielsen’s ease of use

• Error Frequency/User Satisfaction– Quantitative analysis– Qualitative analysis

What is Usability Testing?

• A usability test is a structured process used to explore the interaction between an objective participant and a proposed design

• Paper and pencil (paper prototype)– Heuristic evaluation– Cognitive walkthrough

• Usability Lab (functional prototype)– Formal testing with subjects

What is Usability Testing?

• Attributes common to most usability tests:– The goal is to improve a product.– Participants are real users.– The participants do real tasks.– Participants are formally observed.– The data are analyzed.– Recommendations for improvement are made.

What is Usability Testing?

• A usability test has three basic components: – Participants—Actual users who are asked to

perform realistic and representative tasks using a proposed design

– Design—May be a fully functioning prototype or a simple paper prototype

– Tester—There might be only one tester or there might be a testing team

What is Usability Testing?

• Constraints on Usability Testing– Time

• Design

• Prepare

• Administer (an hour to an hour and a half)

• Analyze the results

– Finance • Equipment and software

• Laboratory time

• Recording media

• Participant compensation

• Refreshments

What is Usability Testing?

• Constraints on Usability Testing– Personnel—Formal usability tests require at

least four people. – Laboratory—To perform a formal usability test, a

dedicated laboratory is required.

What is Usability Testing?

• Advantages– Minimize help desk calls– Increase product loyalty– Provide benchmarks for future products

• Limitations– Artificial context– Not definitive of product acceptance– Skewed sample of users– Not always efficient

What is Usability? – Phases of a Usability Test

• Design the Test

• Prepare for the Test

• Perform the Test

• Process the Data

Design the Test

• Why: Define the Purpose

• What: Define Your Concerns and Goals

• What: Define the Tasks

• What: Create the Scenarios

• What: Define the Measurements

• How: Define the Test Method

Design the Test

• Where: Determine the Location of the Tests

Design the Test

• Who: Select Participants, Testers, and Observers

Design the Test

Participants should be real users

You do not always need to test a great many users

• The people you recruit should have the following basic characteristics:– Availability– Responsiveness– Objectivity

Design the Test

• Tester roles include the following:– Administrator– Moderator– Data logger– Technician– Prototype expert

• Potential observers include the following:– Other design team members not involved in the test– Clients– Programmers responsible for the final product

Prepare for the Test

• When: Create a Test Schedule– Project level– Test preparation level– Test execution level– Task execution level

Prepare for the Test

• Writing Scripts– Greeting the Participant– Preliminary Interview– Providing Instructions– Monitoring the Test– Debriefing the Participant

• Running a Pilot Test– Be organized– Be presentable

Perform the Test

• Test Phases– Pre-Test

• Greet the participant.• Have the participant sign the informed consent form.• Have the participant fill out any pre-test questionnaire.• Proceed with scripts.

– During the Test• Maintain a log or observation check list for each task.• Create a problem list to capture anything that is not

covered by the check list.• Notate problems and jot down any hypotheses that occur

to you about the problems.

Perform the Test

• Test Phases– Post-Test

• Debrief the participant. – post-test questionnaire– verbal interview

• Thank the participant and provide compensation.• Collect, summarize, and organize test data.• Reset the room for the next participant.

Process the Data

• Activities Performed on the Day of the Test– Collecting Data– Summarizing Data– Organizing the Material

• Follow-Up Activities– Categorizing– Analyzing

• Quantitative Data• Qualitative Data

Process the Data• Documenting

– Identify problems• Severity• Frequency• Errors of omission

– Prioritize problems– Theorize reasons– Theorize solutions– Identify successes– Identify areas of uncertainty

Heuristic Evaluation

Heuristic Evaluation• Introduction to Heuristic Evaluation • Phases of Heuristic Evaluation • How to perform the Heuristic

Evaluation• Heuristics• Examples• Severity Rating• Debriefing• Summary

Heuristic Evaluation

• Developed by Jakob Nielsen• Helps find usability problems in a UI design• Small set (3-5) of evaluators examine UI

– independently check for compliance with usability principles (“heuristics”)

– different evaluators will find different problems– evaluators only communicate afterwards

• findings are then aggregated

• Can be performed on working UI or on sketches

Why Multiple Evaluators?

• Every evaluator doesn’t find every problem

• Good evaluators find both easy & hard ones

Heuristic Evaluation• Evaluators goes through UI several times

– inspects various dialogue elements– compares with list of usability principles– consider any additional principles or results that come

to mind

• Usability principles– Nielsen’s “heuristics”– supplementary list of category-specific heuristics

Phases of Heuristic Evaluation

1) Pre-evaluation training– give evaluators needed domain knowledge

and information on the scenarios

2) Evaluation– individuals evaluate and then aggregate

results

3) Severity rating– determine how severe each problem is

(priority)

4) Debriefing– discuss the outcome with design team

How to Perform Evaluation• At least two passes for each evaluator

– first to get feel for flow and scope of system– second to focus on specific elements

• If system is walk-up-and-use or evaluators are domain experts, then no assistance needed– otherwise one might supply evaluators with

scenarios

• Each evaluator produces list of problems– explain why with reference to heuristic or other

information– be specific and list each problem separately

Heuristics• H1-1: Simple and natural dialog• H1-2: Speak the users’ language• H1-3: Minimize users’ memory load• H1-4: Consistency• H1-5: Feedback• H1-6: Clearly marked exits• H1-7: Shortcuts• H1-8: Precise and constructive error

messages• H1-9: Prevent errors• H1-10: Help and documentation

Heuristics

• H2-1: Visibility of system status– keep users informed about what is going on– example: pay attention to response time

• 0.1 sec: no special indicators needed • 1.0 sec: user tends to lose track of data • 10 sec: max. duration if user to stay focused on 1 action • for longer delays, use percent-done progress bars

Mac desktop Dragging disk to trash

should delete it, not eject it

H2-2: Match between system and real world speak the users’ language follow real world conventions

Heuristics

Heuristics

Wizards must respond to Q1

before going to next for infrequent tasks

modem config. not for common tasks

Good for beginners have 2 versions

WinZip

H2-3: User control and freedom “exits” for mistaken choices, undo, redo don’t force down fixed paths

Heuristics• H2-4: Consistency & standards

Heuristics

MS Web Pub. Wiz. Before dialing

asks for id & password When connecting

asks again for id & pw

H2-5: Error prevention H2-6: Recognition rather than recall

make objects, actions, options, and directions visible or easily retrievable

Heuristics• H2-7: Flexibility and efficiency of use

– accelerators for experts (e.g., gestures, kb shortcuts)– allow users to tailor frequent actions (e.g., macros)

• H2-8: Aesthetic and minimalist design– no irrelevant information in dialogues

Heuristics

• H2-9: Help users recognize, diagnose, and recover from errors– error messages in plain language– precisely indicate the problem– constructively suggest a solution

Heuristics

• H2-10: Help and documentation– easy to search– focused on the user’s task– list concrete steps to carry out– not too large

Examples• Can’t copy info from one window to another

– violates “Minimize the users’ memory load” (H1-3)– fix: allow copying

• Typography uses mix of upper/lower case formats and fonts– violates “Consistency and standards” (H2-4)– slows users down– probably wouldn’t be found by user testing– fix: pick a single format for entire interface

Severity Rating

• Used to allocate resources to fix problems • Estimates of need for more usability efforts• Combination of

– frequency– impact– persistence (one time or repeating)

• Should be calculated after all evaluations are in• Should be done independently by all judges

Severity Ratings (cont.)

0 - don’t agree that this is a usability problem

1 - cosmetic problem

2 - minor usability problem

3 - major usability problem; important to fix

4 - usability catastrophe; imperative to fix

Severity Ratings Example

[H1-4 Consistency] [Severity 3][Fix 0]

The interface used the string "Save" on the first screen for saving the user's file, but used the string "Write file" on the second screen. Users may be confused by this different terminology for the same function.

Debriefing

• Conduct with evaluators, observers, and development team members

• Discuss general characteristics of UI• Suggest potential improvements to address

major usability problems• Add ratings on how hard things are to fix• Make it a brainstorming session

– little criticism until end of session

Summary• Heuristic evaluation is a discount method• Have evaluators go through the UI twice• Ask them to see if it complies with heuristics

– note where it doesn’t and say why

• Combine the findings from 3 to 5 evaluators• Have evaluators independently rate severity• Discuss problems with design team• Alternate with user testing