User-Testing, Testing, 1,2,3

the W’s

What?

• What is User-Testing?

User-Testing is a focal part

of the User Centered Design

philosophy that aims to gain

direct feedback from users and

collect actionable data

What?

• What Can We Test?

•Wireframes

•Designs

•Prototypes

•Live Websites

•Everything!

Why?

• Why Test Users?

• We often think we know what is best for our users

• You want to add innovative functionality to your site

• You have low conversion rates

• You have high customer support costs

When?

Benchmark

Test User Expectations

Test Visual Affordance

Test User Performance

Test Live Website

Test Expectations / Mental Model

Who?

• Users, not Testers

• Real people who use your

website

• Rinse, Lather, REPEAT!

• Who Do We Test?

Who?

• Who Do We Listen

To? We need to place four ad

banners on the homepage

I want to read content

that is fresh and

interesting

We need to drive users

to download our latest

whitepaper

I don’t want my

decisions manipulated

We want to display

company news and our

stock ticker on the

homepageI know what I want and

want to find it

without being

distracted

Where?

• Where Do We Find Users to Test?

•On your website

•Through a recruiter

•In their natural

environment

How?

• How can we avoid having to

figure all this out on our own?

Let the users do

all the hard work

for you!

Types of User-Testing


• Remote

• In Lab

• Automated

Types of User-Testing:

Remote

• Users sit in

their natural

environment

• Users are asked

to carry out

specific tasks

using prototypes

• Their mouse

movements and the

phone

conversation are

recorded for

analysis

Types of User-Testing: In

Lab

• Video camera

records users’

facial expressions

• A screen recorder

records their

screen movements

• A facilitator is in

the room with them

• Observers are in

another room,

usually looking

through a 2 way

mirror

Types of User-Testing:

Automated• Users are

intercepted when they appear on a site

• Users are asked to fill out a short survey before navigating the site

• All the user’s movements are recorded while navigating the site

• Users complete an exit survey when they leave the site


Type Pros Cons What to Use it On

Remote Nationally diverse geographic reach

Quick

Less Costly

Lose the ability to see users’ expressions and gestures

Unnatural environment

E-commerce web sites

Large, informational web sites

Web Applications

Intranets

In Lab Obtain data based on user’s body movement and facial expressions

More costly

Lack of geographic reach

Highly secure client/server applications

Handheld Apps or other products with a significant hardware component

Automated Captures users that come to the site and records users’ activities

Natural environment

Actual users

Steps in between start and goal is not elicited

E-commerce web sites

Large, informational web sites

Web Applications

Intranets

Types of Data Collected

• Quantitative

– Statistics,

Trends

– Large Sample Size

– Structured Data

Collection

Methods

• Qualitative

– Opinions,

Details

– Small Sample

Size

– Unstructured

Data Collection

MethodsExample:

―I tried clicking on the button

because the white text on the

blue background caught my eye

and the text was large‖

Example:

•30% of users clicked on the red

button

•70% of users clicked on the blue

button

Informal vs. Formal

Testing

• Informal

– Casual

– 3-4 Days

– In Your Office

– Friends & Family

– Paper Prototypes

– Less Common

– Bias

– Incorrect

Analysis

• Formal

– Structured

– 3-4 Weeks

– In a Lab

– Broad Range of

Users

– High Fidelity

– More Common

– Trained Moderators

& Analysts

POLL

• Have you conducted any type of

User-Testing on your current

website?

How User-Testing Fits in

the Web Development

Process

The Web Development Process

Benchmark

Test User Expectations

Test Visual Affordance

Test User Performance

Test Live Website

Test Expectations / Mental Model

The Testing Methods

Persona Development

Wireframe Testing

Visual Affordance

Testing

User Acceptance

Testing

Multivariate Testing

Card Sorting

Persona Development (PD)

PD:

Persona Development

Wireframe Testing

Visual Affordance

Testing

User Acceptance

Testing

Multivariate Testing &

Optimization

Card Sorting

PD:

How do we develop a website that

is tailored to our users needs,

behaviors, experience, and

technology?

Ask

them!

Q:

A

:

PD: What is a Persona?

• A persona tells the story about the users of your website and attributes lifelike features to make them more relatable

• Based on real data, the persona serves as a tool to help everyone involved in development understand who and what the website is for

• The persona creates a shared vision

PD: Market Segments vs.

Personas

Market Segment

• Gender: 75% Female, 25% Male

• Region: 99% in United States

• Size: 70% metro, 20% suburban, 10% rural

• Generation: 10% baby-boomers, 20% Gen-X, 70% Gen-Y

• Income: 25k – 50k

• Occupation: Student, Administrative Assistant,

Intern

• Education: High School Diploma

• Family Lifecycle: Single

PD: Market Segments vs.

Personas

• Persona

PD: Surveys & Interviews

Surveys

• Quantitative Method

• Need about 400

responses to account

for margin of error

Interviews (contextual +

individual)

• Qualitative Method

• Need about 8 participants

per segment

PD: Benefits

• Focus on the users’ goals and

needs throughout the development

process

• Prioritize features, design,

efforts based on what is most

needed by the personas

• Any internal battles will can be

settled by referring to the

personas

• Develop a site that users will

PD: Tips for Success

• Always include:

– A Name

– A Photo

– Demographics

– Description based on data collected

– Goals

– Common tasks carried out on the website

– Habits

• Always refer back to the personas!

Information Architecture

Testing (IAT)

IAT: Card Sorting

Persona Development

Wireframe Testing

Visual Affordance

Testing

User Acceptance

Testing


Optimization

Card Sorting

IAT: Card Sorting

How do we ensure that the

taxonomy and site linking

structure are intuitive for our

users to find our products and

services?

Have the user label

and organize pages

for you!

A

:

Q:

IAT: What is Card Sorting?

• Users are given ―cards‖ with

labels on them and asked to

categorize the information on

your site into meaningful

groupings.

• Method: In-Lab or Automated

• Sample Size: At least 15 per

segment

IAT: Types of Card Sorting

Open Card Sorting

• Categories and category labels can be created or changed by users

• Used early during the development cycle

• More difficult to analyze data

Closed Card Sorting

• Fixed categories are provided to the user and they may not re-label or created new categories.

• Used later in development cycle for further testing, or to add new content to the website

• Easier to analyze data statistically

IAT: Card Sorting Tools

(Lo-Fi)

IAT: Card Sorting Tools

(Online)

• Online tools allow for easier

data collection / analysis and

larger sample sizes

POLL

• What is your primary method for

conducting card sorting?

IAT: Card Sorting Benefits• Easy to administer

• Sets the foundation for the website’s navigation

and structure

• Intuitive taxonomy in the users’ ―natural

language,‖ and matches what users are searching for

(SEO)

• Intuitive taxonomy in the users’ ―natural

language,‖ and matches what users are searching for

(SEO)

IAT: Card Sorting Example

IAT: Card Sorting Tips for

Success

• If in-lab, ask users to talk out

loud

• Limit the study to a reasonable

amount of cards (~50)

• Note the relationships between

cards and situations that indicate

cross-linking between categories

will be helpful for users

IAT: Wireframe Testing

Persona Development

Wireframe Testing

Visual Affordance

Testing

User Acceptance

Testing


Optimization

Card Sorting


How do we find out if the layout

of information on the site is

intuitive, and if the placement of

images, text, links, messaging, and

calls-to-action help users find

what they need?

Watch users as they

navigate through

the wireframes!

Q:

A

:

IAT: What is Wireframe

Testing?

• Wireframe testing is a usability test of

the prototype pages of your site

excluding colors and other details. This

involves creating realistic tasks and

having users navigate the prototype to

fulfill those tasks.


• Method: In-Lab or Remote


segment

– Why 5 Users?


Example

• You are out in the

field and you need to

find information on

how to troubleshoot

the network. Where

would you start?

• Your business has just

switched over to VoIP.

Where would you go to

look for information

on how to manage your

new VoIP network?

IAT: Wireframe Testing:

Benefits

• Find out if the layout of the page

achieves your users’ goals and your

goals

• Low cost and quick to make changes

to a wireframe versus a real

website

IAT: Wireframe Testing: Tips

for Success

• Create tasks that are real

situations for the user (refer to

the personas)

• Do not bias the participant by

leading or priming

• Be patient!

Visual Affordance Testing

(VAT)

VAT:

Persona Development

Wireframe Testing

Visual Affordance

Testing

User Acceptance

Testing


Optimization

Card Sorting

VAT:

How do we make sure the design is

not only aesthetically pleasing,

but also that messaging and calls-

to-action are clear for the user?

Ask users, and

watch as they

navigate through

the designs!

Q:

A

:

VAT: What is Affordance?

• Affordance is a quality of an

object, or an environment, that

allows an individual to perform

an action.

• A visual clue to the function

of an object

VAT: What is Visual

Affordance Testing?

• Once images, colors, and other

design elements have been

implemented based on the

wireframes, VAT requires users to

indicate what they perceive as

clickable on the design


• Sample Size: At least 5 per segment

VAT: Benefits

• Validates if key calls-to-action

are even seen by users

• Reveals elements of a design which

may mislead users to believe are

clickable

• Reduce user frustration by

addressing problems found in the

VAT

VAT: Example

VAT: Tips for Success

• Pay close attention to any

hesitation or vocal expressions

• Visual affordance does not only

apply to calls-to-action or links

User Acceptance Testing

(UAT)

UAT:

Persona Development

Wireframe Testing

Visual Affordance

Testing

User Acceptance

Testing


Optimization

Card Sorting

UAT:

How do we find out if all the

content, interactions, and media

on the prototype we’ve built will

resonate with our visitors, and if

they negatively affected our

site’s usability?

Ask users, and watch as

they navigate through the

prototype!

Q:

A

:

UAT: What is User Acceptance

Testing?

• User Acceptance Testing is a

usability test conducted on a

fully functional prototype

prior to launching the website



segment

UAT: Benefits

• Find out how users will really

use the website

• Figure out if all functionality

and interactivity is useful to

the user (Flash, Videos, AJAX)

• Observe where people may have

difficulty or fail and how they

recover from those errors

UAT: Tips for Success

• User Acceptance Testing is not

Quality Assurance!

• Allow for free exploration with

the user thinking aloud

• Test iteratively

POLL

• Have you added any new content

to your website in the past 6

months (whitepapers, downloads,

videos)?

Multivariate Testing (MVT)

MVT:

Persona Development

Wireframe Testing

Visual Affordance

Testing

User Acceptance

Testing


Optimization

Card Sorting

MVT:

How do we make sure that our

site is always usable and that the

user-experience of our visitors is

always optimal?

Continue to test and

optimize your site!

Q:

A

:

MVT: What is Multivariate

Testing?

• Once the website is launched,

Multivariate Testing optimizes

performance of the website by

segmenting visitors and displaying

different images/ buttons/ links/

content to them

• Method: Automated, In-Lab, Remote

• Sample Size: At least 500 visitors

MVT: A/B Testing vs.

Multivariate Testing

A/B MVTLanding Page Version 1 Landing Page Version 2

Views 185,854 187,900

Form Submissions 4,154 5,232

Conversion Rate 2.24% 2.78%

Landing Page Element Views Form Submissions Conversion Rate

1C - Banner [static image] 1,300 517 39.77%

2A - Button ["Buy Now"] 1,287 270 20.98%

3B - Text [black] 1,340 245 18.28%

2C - Button ["Submit"] 1,310 230 17.56%

1B - Banner [AJAX] 1,305 125 9.58%

3A - Text [gray] 1,290 105 8.14%

3C - Text [blue] 1,297 105 8.09%

2B - Button ["Next"] 1,304 99 7.60%

1A - Banner [Flash] 1,289 88 6.83%

MVT: Benefits

• Minimize risk with any website

changes

• Data drives decisions

• Data is collected 24/7

• Tangible Results

• Test New Applications

• Test the ―Third Screen‖

MVT: Tips for Success

• ABT – Always Be Testing!

• Preparation and set-up is

crucial

What Next?

• Rinse, Lather, REPEAT!

―If ease of use was the only valid

criterion, people would stick to

tricycles and never try bicycles‖Doug Engelbart

Human-Computer Interaction Pioneer

and inventor of the computer mouse

Thank You!Jegan Chen, Usability Engineer

BusinessOnLine

[email protected]

http://www.businessol.com

Additional BusinessOnLine

Resources:

Re:Cognition: BusinessOnLine’s Usability Blog

http://www.businessol.com/usability_blog

Upcoming Webinar:

SEO Webinar Series:

Session 3: Not Just Video—VideoSEO

June 17, 2009 11am PST (2pm EST)

http://www.businessol.com/news/main-webinars/

mailto:[email protected]

http://www.businessol.com/

http://www.businessol.com/usability_blog




User-Testing, Testing, 1,2,3

Self Improvement

Transcript of User-Testing, Testing, 1,2,3