User Interface Evaluation

Usability Testing Methods

Usability Testing Methods Conduct experiments to find specific information

about a design and/or product.

Basis comes from experimental psychology.

Uses statistical data methods Quantitative and Qualitative

Usability Testing Methods During usability testing, users work on specific

tasks using the interface/product and evaluators use the results to evaluate and modify the interface/product.

Widely used in practice, but not appropriately used.

Often abused by developers that consider themselves to be usability experts.

Can be very expensive and time consuming.

Usability Testing Methods

Performance Measurement

Thinking-aloud Protocol

Question-asking Protocol

Coaching Method

Usability Testing Methods

Co-discovery Learning

Teaching Method

Retrospective Testing

Remote Testing

Performance Measurement

Applicable Stages: Design, Code, Test & Deployment

Personnel Usability Experts, approximately 1. Developers, 0. Users, 6.

Performance Measurement Usability Issues Covered

Effectiveness: Yes Efficiency: Yes Satisfaction: No

Quantitative Data is collected.

Can NOT be conducted remotely.

Can be used on any system.

Performance Measurement What is it?

Used to collect quantitative data.

Typically, you will be looking for benchmark data.

Objectives MUST be quantifiable 75% of users shall be able to complete the basic task

in less than 30 minutes.

Performance Measurement How can I do it?

Define the goals that you expect users to perform

Performance Measurement How can I do it?

Quantify the goals The time users take to complete a specific task. The Ratio between successful interactions and errors. The time spent recovering from errors. The number of user errors. The number of commands or other features that were

never used by the user. The number of system features the user can remember

during a debriefing after the test. The proportion of users who say that they would prefer

using the system over some specified competitor.

Performance Measurement How can I do it?

Get participants for the experiments

Conduct very controlled experiments All variables must remain consistent across users

Performance Measurement Problems With Performance Measurement

No qualitative data.

Thinking-aloud Protocol Applicable Stages:

Design, Code, Test & Deployment

Personnel Usability Experts, approximately 1. Developers, 0. Users, 4.

Thinking-aloud Protocol Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Thinking-aloud Protocol What is it?

Technique where the participant is asked to vocalize his or her thoughts, feelings, and opinions while interacting with the product. .

Thinking-aloud Protocol How can I do it?

Select the participants, who will be involved?

Select the tasks and design scenarios.

Ask the participant to perform a task using the software.

Thinking-aloud Protocol How can I do it?

During the task, ask the user to vocalize Thoughts, opinions, feelings, etc.

http://www.cs.umd.edu/~zzj/ThinkAlo.htm

Thinking-aloud Protocol Problem With Thinking-Aloud Protocol

Cognitive Overload Can you walk & chew gum at the same time? Asking the participants to do too much.

Question-asking Protocol Applicable Stages:

Design, Code, Test & Deployment

Personnel Usability Experts, approximately 1. Developers, 0. Users, 4.

Question-asking Protocol Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Question-asking Protocol What is it?

Similar to Thinking-aloud protocol.

Instead of participant saying what they are thinking, the evaluator prompts the participant with questions while using the system.

Question-asking Protocol How can I do it?

Select the participants, who will be involved?

Select the tasks and design scenarios.

Ask the participant to perform a task using the software.

Question-asking Protocol How can I do it?

During the task, ask the user to questions about the product Thoughts, opinions, feelings, etc.

http://www.cs.umd.edu/~zzj/Question.htm

Question-asking Protocol

Problem With Thinking-Aloud Protocol

Cognitive Overload++ Can you walk, chew gum & talk at the same time? Asking the participants to do too much. Added pressure when the evaluator asks questions. Can be frustrating on novice users.

Coaching Method Applicable Stages:

Design, Code, Test & Deployment

Personnel Usability Experts, approximately 1. Developers, 0. Users, 4.

Coaching Method Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Coaching Method What is it?

A system expert sits with the participant and acts as a coach.

Expert answers the participant’s questions.

The evaluator observes their interaction.

Coaching Method How can I do it?

Select the participants, who will be involved?

Select the tasks and design scenarios.

Ask the participant to perform a task using the software in the presence of a coach/expert.

Coaching Method How can I do it?

During the task, the user will ask the expert questions about the product

http://www.cs.umd.edu/~zzj/Coaching.htm

Coaching Method Problem With Coaching Method

In reality, there will not be a coach present.

This is good for creating a coaching system, but not for evaluating the interface.

Co-Discovery Learning

Applicable Stages: Design, Code, Test & Deployment

Personnel Usability Experts, approximately 1. Developers, 0. Users, 6.

Co-Discovery Learning Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Co-Discovery Learning What is it?

Two test users attempt to perform tasks together while being observed.

They are to help each other in the same manner as they would if they were working together to accomplish a common goal using the product.

They are encouraged to explain what they are thinking about while working on the tasks. Thinking Aloud, but more natural because of partner.

Co-Discovery Learning How can I do it?

Select the participants, who will be involved?

Select the tasks and design scenarios.

Ask the participants to perform a task using the software.

Co-Discovery Learning How can I do it?

During the task, the users will help each other and voice their thoughts by talking to each other.

http://www.cs.umd.edu/~zzj/Codiscov.htm

Co-Discovery Learning Problem With Co-Discovery Learning

Neither is an expert The blind leading the blind.

Teaching Method Applicable Stages:

Design, Code, Test & Deployment

Personnel Usability Experts, approximately 1. Developers, 0. Users, 4.

Teaching Method Usability Issues Covered

Effectiveness: Yes Efficiency: No Satisfaction: Yes

Quantitative Data is NOT collected.

Can NOT be conducted remotely.

Can be used on any system.

Teaching Method

What is it?

You have 1 participant use the system.

Ask the participant to teach a new novice participant how to use the system.

Teaching Method

How can I do it?

Select the participants, who will be involved?

Select the tasks and design scenarios.

Ask the 1st participant to perform a task using the software.

Ask the 1st participant to teach a new participant.

Teaching Method How can I do it?

Observe their interactions.

http://www.cs.umd.edu/~zzj/Teaching.htm

Teaching Method Problem With Teaching Method

Neither is an expert The blind leading the blind. Possible to discover some interesting things about the

learn-ability of your interfaces.

Retrospective Testing Applicable Stages:

Design, Code, Test & Deployment

Personnel Usability Experts, approximately 1. Developers, 0. Users, 4.

Retrospective Testing Usability Issues Covered

Effectiveness: Yes Efficiency: Yes Satisfaction: Yes

Quantitative Data can be collected.

Can NOT be conducted remotely.

Can be used on any system.

Retrospective Testing What is it?

A videotape of the session is observed by the usability expert and the participants.

Retrospective Testing How can I do it?

Select the participants, who will be involved?

Select the tasks and design scenarios.

Use one of the usability testing methods that we have discussed.

Videotape the session.

Retrospective Testing How can I do it?

Review the videotape with the users.

http://www.cs.umd.edu/~zzj/Retrospe.htm

Retrospective Testing Problem With Retrospective Testing

Extremely time consuming!

Remote Testing Applicable Stages:

Design, Code, Test & Deployment

Personnel Usability Experts, approximately 1. Developers, 0. Users, 5.

Remote Testing Usability Issues Covered

Effectiveness: Yes Efficiency: Yes Satisfaction: Yes

Quantitative Data can be collected.

Can be conducted remotely.

Can be used on any system.

Remote Testing

What is it?

The participants are separated from the evaluators.

No formal observation.

No usability lab.

Remote Testing How can I do it?

Give the product/software to participants.

Collect information about how they use your software/product.

Methods Same-Time Different Place Different-Time Different Place

Remote Testing How can I do it?

Camtasia, Look@Me, SnagIt

Usability Logger http://www.usabletools.com/

Journaled Sessions

http://www.cs.umd.edu/~zzj/Retrospe.htm

Remote Testing

Problem With Remote Testing

The evaluator is not there.

Can’t observe facial expressions.

Great for Web based systems.

Usability Testing MethodsSelect the method that works best for you.

Select the method that fits your implementation.

Be thorough during your experiments.

The more data, the better.

Usability Testing MethodsHawthorne Effect

The tendency for people to improve their performance after any change when they know their performance is being studied.