Exploring Usability Testing

TI Half-day Tutorials

5/6/2014 8:30:00 AM

Exploring Usability Testing

Presented by:

Rob Sabourin

AmiBug.com

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Rob Sabourin AmiBug.com

Rob Sabourin, P. Eng., has more than thirty years of management experience leading teams of software development professionals. A well-respected member of the software engineering community, Rob has managed, trained, mentored, and coached hundreds of top professionals in the field. He frequently speaks at conferences and writes on software engineering, SQA, testing, management, and internationalization. Rob wrote I am a Bug!, the popular software testing children's book; works as an adjunct professor of software engineering at McGill University; and serves as the principle consultant (and president/janitor) of AmiBug.Com, Inc. Contact Rob at Contact Rob at [email protected].

© 2013 Robert Sabourin 1

© 2013 Robert Sabourin EUT v1.1 1


Robert Sabourin

President

AmiBug.Com, Inc.

Montreal, Canada

[email protected]

WELCOME




• Robert Sabourin ,

Software Evangelist

• President

• AmiBug.Com Inc.

• Montreal, Quebec,

Canada

• [email protected]

• www.amibugshare.com




• Pain points?

– What hurts?

– How Much?



SOME PHILOSOPHY



Conforming to requirements

Suiting purpose

Delivering value to stakeholders



Edsger W. Dijkstra

• “Program testing can be used to show the

presence of bugs, but never to show their

absence”


NON-FUNCTIONAL

TESTING




Non-Functional Testing

Quality Factors

Attributes

Characteristics

Reliability

Other “-ilities”


Non-Functional Testing Challenges

Goals

• Elicit

• Articulate

• Quantify

Subjective

• Needs or wants?

• Relative or absolute?

• What is good enough?

Tests

• Tricky to orchestrate

• Difficult to interpret results

• Challenging to baseline or regress

10© 2013 Robert Sabourin EUT v1.1


Non-Functional Agile Challenges

Time

• Analysis

• Model

• Prepare

• Run

• Interpret

Change

• Baselines

• Goals

Software

• Continuous integration

• Incomplete product builds


ISO 9126 Quality Factors

•Attributes of the effort needed for use, and on the individual assessment of such use, by a set of users

•Learnability

•Understandibility

•Operability

Usability



Quality Factors


Product Backlog Constraints

Tom Gilb, Competitive Engineering, suggests defining

Scale: "What is measured"

Meter: "How to measure (method)"

Target: "Level we're aiming for. Success"

Constraint: "Level we're seeking to avoid. Failure"

Benchmark: "Where we are today"



Non-Functional Testing


NET PROMOTER SCORE




Net Promoter Score


Net Promoter Score



Net Promoter Score

•11 values

•0 not at all likely

•10 extremely likely

Response Scale


Net Promoter Score

• respondents giving a 9 or 10 score

Promoters

• respondents giving a 7 or 8 score

Passives

• respondents giving a 0 to 6 score

Detractors



Net Promoter Score

•The difference between the percentage of Promoters and Detractors

• Is not expressed as a percentage

• Is an absolute number lying between -100 and +100

•uTest suggests “any positive number is good” (ref: TK Maxx Mobile Usability Testing Results Report)

NPS


Net Promoter Score




SYSTEM USABILITY SCALE




System Usability Scale

•Measures usability in terms of effectiveness, efficiency and satisfaction

• SUS can be read as a percentage

• SUS can be used as a benchmarking mechanism

• SUS can be used to compare the current system to competitors’ systems

• SUS can be used to compare the current system to future versions

SUS - System Usability Scale






1. I think that I would like to use this system frequently2. I found the system unnecessarily complex3. I thought the system was easy to use4. I think that I would need the support of a technical

person to be able to use this system5. I found the various functions in this system were well

integrated6. I thought there was too much inconsistency in this

system7. I would imagine that most people would learn to use

this system very quickly8. I found the system very cumbersome to use9. I felt very confident using the system10. I needed to learn a lot of things before I could get

going with this system


SUS Questions



SUS Responses



• For odd items: subtract one from the user response.

• For even-numbered items: subtract the user responses from 5

• This scales all values from 0 to 4 (with four being the most positive response).

• Add up the converted responses for each user and multiply that total by 2.5. This converts the range of possible values from 0 to 100 instead of from 0 to 40.


Scoring SUS



Scoring SUS


EXPLORATORY TESTING



Exploratory Testing

© 2013 Robert Sabourin EUT v1.1

Learning

DesignExecution

32


Charter Statement

• Statement of mission

• Ties to purpose

• Focuses work

• Confirms understanding

• Delineates scope

• Analogy to test story


Charter Statement

• Short, terse

• To the point

• Inclusions /

• Exclusions /

• Limits /



eBay Usability Charters





Pairing Testers With …

Non Functional Testing

•Usability Gurus

•SMEs

•Human Factors Experts


Heuristics



Heuristics

Guide Exploration

Rules of thumb

Fallible but useful


WHITEBOARDING




Prototype Styles

• Paper and pencil• Comic books style story boards• Wire frame models• Mock ups• Prototypes• Site maps• User flow• Wire flow• Story boards• Wire frame• Page description diagram• Functional specification


Whiteboarding



Whiteboarding

What is Whiteboarding?

Collaboration

Coordination

Conceptualization


Tony Buzan

Mind Maps



Whiteboarding


Whiteboarding

User Mind Map



Buying a Book

Usage Scenarios


Whiteboarding

Story Board



Wrap-O-Matic

Usage Scenarios


Scenario Based Testing



Buying a Book

Usage Scenarios



Slide 54



Slide 55



Social Networking

Usage Scenarios




TOFT Testing

• Task Oriented

Functional Testing

– Can the user

accomplish useful

tasks correctly?

© 2013 Robert Sabourin EUT v1.1 Slide 67

USABILITY HEURISTICS




Jakob Nielsen

• Usability heuristics

• Rules of thumb

• General principles of user interface design

• www.nngroup.com


Usability Heuristics

• Visibility of system status

– Always keep users informed

– Provide appropriate feedback

– Respond in a reasonable timeframe




• Match between system and the real world

– Speak the users' language

– Use familiar concepts

– Follow a natural order



• User control and freedom

– Provide means to exit from unwanted states

– Provide undo and redo capabilities




• Consistency and standards

– Same words mean same thing in different contexts

– Follow environment conventions



• Error prevention

– Prevents problems from occurring

– Eliminate error-prone conditions

– Have a confirmation option




• Recognition rather than recall

– Making objects visible

– Make instructions available



• Flexibility and efficiency of use

– Accelerators for the expert user

– Make common options adaptable




• Aesthetic and minimalist design

– Dialogues should contain relevant information



• Help users recognize, diagnose, and recover from errors

– Use plain language for error messages

– Constructively suggest solutions




• Help and documentation

– Easy to search

– Focused on the user's task

– Offer concrete steps to be carried out

– Short and to the point


USABILITY SEVERITY

LEVELS




Usability Severity


ISO 9421




Usability ISO 9421

•the extent to which the intended goals of use are achieved

Effectiveness

•the resources that have been expended to achieve the intended goals

Efficiency

•the extent to which the user finds the use of the product acceptable

Satisfaction

•can user be harmed

Safety


Usability ISO 9421

•the extent to which the system is complete in design and meets its desired goals

Completeness

•Is the capability of a software product to enable the user to learn how to use it

Learn-ability

•the extent to which the product fits into normal course of daily life

Convenience

•Confusing or vague usability. Misleading information that might also result in providing no usability at all

Ambiguity



USABILITY TEST

FRAMEWORK



Usability Test Framework

Basic Usability Test Steps

• Select representative users

• Complete defined set of tasks

• Facilitator observes user behaviour

• Interview user after task is completed

• User is asked to think aloud (recorded)

• Review findings to identify concerns

• Generate actionable recommendations




User groups

•Select 5 or 6 users for each group

•All should attempt the same tasks

•Complex applications may need many groups

•Groups can be based

•Domain (subject matter) expertise

•Experience level (task or professional)

•Solution familiarity



Where to perform usability tests?

• Control lab

• Conference room with recording equipment

• Work area with

• Local

• Distributed with local observer

• Distributed without local observer




Prepare Usability Scenario

• Identify project usability test objectives

•Select relevant user tasks

•Keep user instructions short and to the point

•Do not over complicate instructions

•Use the language of the user not the language of the system

•Make task parameters specific

•Use a logical flow



Running a test

•Let participants make mistakes

•Take note of questions

•Take note of decisions points

•Take note of alternative selections




Think Aloud Approach

• the user verbalizes actions and decisions

• user verbalizes questions concerns or ambiguities

Retrospective Think Aloud

• records and eye tracking software record event

• review with user and ask them to explain their actions



Reporting Findings

• List of usability concerns

• Summary of user questions

• Description of users behaviour

• Tasks of concern

• Too long

• Awkward

• Unsuccessful

• Time to complete

• Feedback

• Expected

• Unexpected




Measures and metrics

•Effectiveness

•Efficiency

•Satisfaction

•Error frequency

•Memorability



Measures and metrics

• Successful task completion

• Error rates

• Time on task

• Subjective comments

• Likes

• Dislikes

• Recommendations




Facilitating a test

• Make participant comfortable with environment and technology

• Give participants a change to ask questions

• Explain process

• Ask participants about the amount of experience they have with the technology

• Ask participants about the amount of experience they have with the domain

• Review task description with participant

• Explain facilitators involvement


FORMS AND WORKFLOWS




Form Design Principles

• Minimize pain• Illuminate path• Consider context• Ensure consistent communication


Luke Wroblewski

Web Form Design

Eye Tracking



Form Usability

• Number and location of errors• Severity of errors• Completion rates• Time to complete forms• Satisfaction scores• Subjective comments


Awkward Form Path



Clear Form Path


Form Usability

• Eye tracking– What people looked at

– Number of eye fixations

– Length of eye fixations



Form Usability Checklist

�Any unnecessary questions�Succinct questions�Sincere voice�Natural language�Logical groups of information�Structure form as a conversation�Ask optional questions separately


Form Usability Checklist

�Name the form�Name sections of the form�Clear scan line�Spacing between sections�Minimal distractions�Start page guidance for long forms�Indicate progress



Progress Indication


Input Validation

�Suggest valid answers�Illustrate valid format�Validate syntax after user enters data�Indicate limits or ranges of inputs�Provide smart defaults�Hide unneeded controls



Error Messages

�Provide clear error messages�Indicate if error blocks completion�Guide users to resolve error�Provide visual emphasis of error�Use red text and icons for errors�Indicate successful completion


DO IT YOURSELF

USABILITY TESTING




Do it yourself


Do it yourself

A morning a month

Get insights from users

Watch real users



Do it yourself

Recruit diversely

Find important problems

Improve design


Do it yourself

Three testers

Test on site

Observe shared screens



Do it yourself

Observers take notes

User questions recorded

Action decided at lunch


SOME MOBILE USABILITY

CONCERNS




Mobile Usability

•Small Screens

•Limited bandwidth

•Fingers

•No Flash Like GUI

Difficulties


Mobile Usability

•Bad first impression

•Operator errors

•Negative feedback / reviews

•Competitor fills gap

Risks



Mobile Usability

• Important stuff on top

• Easy to find stuff

• Single column layout

• Minimize navigation

• Light data

Guidelines


Mobile Usability

• Finger taps

• Avoid fancy web gui technologies

• Minimize forms

• Minimize feature sets

• Separate mobile web resources

• Fast

Guidelines



Mobile Usability

•In the lab

•Beta

•Crowd source

Mobile Usability Test


Mobile Usability

• Vary and control

• Technologically

• OS

• Browser

• Device

• Carrier

• Bandwidth

Mobile Usability Test



Mobile Usability

•Geographically

• Continent

• Country

•City

• Language

• Locale

Vary and control


Mobile Usability

•Demographically

•Age

•Gender

• Education

•Employment

• Industry

•Computer savvy

•Domain expertise

Vary and control



LEAN UX



Lean UX



Lean UX

•Cross functional teams

•Small teams

•Dedicated teams

•Collocated teams

• Progress equals Outcomes

• Problem focused teams

Lean Principles


Lean UX

•Remove waste

•Small batch size

•Continuous discovery

•GOOB Get out of the building

•Shared understanding

•No Rock Stars Gurus or Ninjas

Lean Principles



Lean UX

•Externalize Your Work

•Making over Analysis

• Learning over Growth

•Permission to Fail

•Getting out of the Deliverables Business

Lean Principles


Lean UX


Assumptions

Hypotheses

Outcomes

Personas

Features


Lean UX


•Business

•User

Assumptions

Lean UX


We believe that creating this experience for these personas will achieve this outcome. We will know this to be true when we see this feedback or measure.

Hypotheses


Lean UX


Outcomes

More business

More referrals

More fun

Lean UX


Personas

• Sketch and Name

•Demographic

• Pain points & needs

• Potential solutions


Lean UX


•Capabilities

•Tactics

•Products

Features are

Lean UX


•Business

•Customer

•User

Features serve


Lean UX


Features drive customer behaviour

Lean UX


Process


Thank You

• Questions?


Exploring Usability Testing

Technology

rob sabourin

software engineering

contact rob

current system

questions

participants

user

efficiency

Transcript of Exploring Usability Testing

Evaluation Assignment 4 Usability Testing Plan...1. Usability Test Plan Usability test plan is required to meet the need of proper structure of usability testing. Usability testing

The Journal of Usability Studies - Exploring the Boundary ...uxpajournal.org/wp-content/uploads/sites/8/pdf/JUS...2000), and usability testing (Sonderegger & Sauer, 2010). Despite

Usability Testing Participants

Guerrilla usability testing

Usability testing

Engaging Users Through Usability Testing · 2018-04-14 · Engaging Users Through Usability Testing. Introduction Middle Tennessee State University, Murfreesboro ... Usability testing

Simple usability testing

Exploring Usability Testing for Mobile and Web Technologies

Usability Testing - Stanford University · Usability Testing November 13, 2019 4 Outline •Why do usability testing? •Choosing participants •Ethical considerations •Designing

Exploring Mobile Usability Testing Methods for Enterprise Users: A Case Study

Usability Testing Materials

Overseas Usability Testing

USABILITY TESTING - Akendi UK · > Reasons to conduct usability testing > Types of usability testing > Pros and cons of expert vs. user-based usability evaluation > Fitting usability

Usability Testing. What is usability testing for?

Medical Device Usability Testing PDF - Kaleidoscopekascope.com/.../08/...to-Medical-Device-Usability-Testing-Success.pdf · •Usability testing is about optimizing the user interface

Usability Testing Infographic

11 Usability Testing

Usability Testing Report

236: II'nMI Usability Testing. What is Usability Testing? Usability testing: What is it? A way to assess the usability of a design with real-world users,

Usability & Usability Testing