The Use of an Unmoderated, Online

Usability Testing Service to Test a Website

Brandon Kopp & Bill Mockovak

Research PsychologistOffice of Survey Methods ResearchUXPA-DC User Focus Conference

19 October 2012

2

TryMyUI

• Provides panel of participants that are presented with user defined tasks

• You receive a video screenshot of the participant’s computer with an audio voiceover of their comments

Many Alternatives

OpenHallway.com

UserTesting.com

Loop11.com

UTest.com

Userzoom.com

OptimalWorkshop.com

The BLS.gov Dropdown Usability Test

• During earlier usability testing, several users had problems with immediately activating dropdown menus

• Users would accidentally click menu links and get lost

Prototypes

• Test design alternatives for mega-dropdowns

Prototype Type of Menu Control Display Width List of Subject Areas

A Click to show/close Fixed, 3 columns Partial list of content, with ‘view all’ option

B Hover, no delay Fixed, 1 column Compact

C Hover, with delay Fixed, 3 columns Current content

D Hover, no delay Fixed, 3 columns Current content

Prototype A

• Need to click on tab to open

• Click on X or Arrow to close

• Abbreviated list is initially presented

• User clicks View All to see entire list

Prototype B

• Dragging mouse over menu will open it

• Menu closes when you move mouse off it (no clicking is necessary)

• Higher-level categories are displayed

Prototype C

• Current, ‘full’ list of topics

• Menu drops after delay, with mouseover

• Moving off menu makes it disappear quickly

Prototype D

• Control Condition• Current menu on

BLS.gov• Menu drops

immediately (no delay) with mouseover

• Moving off menu makes it disappear quickly

Method

• Define Participant Criteria

• Participants complete up to 10 tasks while providing verbal feedback

Gender: Any Age: 18-55

Country: U.S. Income: Any

Education: Any Employment: Any

Computer Experience: Beginner-Intermediate or Expert

Prototype# of

ParticipantsA 11

B 14

C 10

D 10

Tasks

1. Find a publication called the Occupational Outlook Quarterly. This is an online magazine about jobs and careers.

Tasks

1. Occupational Outlook Quarterly2. Mass Layoffs3. International Unemployment4. Green Jobs5. Survey of Occupational Illness and Injury6. Strikes and Lockouts7. Students and Teachers8. NY State Wages

14

What the Participant Sees

Task Instructions

Browser / Webpage View

Recording Time

What The Experimenter Sees

Video of Participant’s Screen

Task Index

Example Videos

Feedback (A; Task 8)

Feedback (D; Task 6)

Task Success and Time To Complete

A B C D0

1

2

3

4

5

6

7

8

6.26.6

7.4

6.7

# of

Tas

ks C

ompl

eted

(out

of 8

)

A B C D0

20

40

60

80

100

120

96.9

83.887.8

77.8

Tim

e to

Com

plet

e Ta

sk (i

n se

c.)

# of Tasks Successfully Completed Time to Complete Task

First Click

Task Home Subject Areas

Databases & Tools

Economic Release Publications Elsewhere

on page

1 0 3 3 0 32 7

2 3 17 16 7 0 2

3 3 26 9 6 0 1

4 2 33 4 2 0 2

5 1 34 6 1 1 0

6 3 38 3 1 0 0

7 4 25 2 1 4 7

8 1 25 9 1 0 6

Method

• Define Participant Criteria

• Participants complete up to 10 tasks while providing verbal feedback

• Following testing, participants write responses to 4 open-ended questions

Gender: Any Age: 18-55

Country: U.S. Income: Any

Education: Any Employment: Any

Computer Experience: Beginner-Intermediate or Expert

Prototype# of

ParticipantsA 11

B 14

C 10

D 10

Open-Ended Questions

Easy/Very Easy Neither Easy Nor Difficult Difficult/Very Difficult0

10

20

30

40

50

60

70

80

47

20

33

61

0

39

67

17 17

69

8

23

A B C D

% o

f Par

ticip

ants

Sel

ectin

g Ra

ting

• How easy were these tasks to complete? Were they very easy, easy, neither easy nor difficult, difficult, or very difficult? How easy do you think these tasks would be for an average American citizen using this website?

Feedback/Recommendations• Final Recommendation:

Prototype C

• Several participants were surprised that the Home tab had a menu

• No items under Workplace Injuries subcategory on Subject Areas menu

Feedback/Recommendations

• Subject Areas menu extended off the screen on some participants’ browsers requiring scrolling

Advantages

Method CostTraditional, In-Lab $40 per participantTryMyUI.com $35 per participantUserTesting.com $39 per participantUTest.com --OpenHallway.com $49-$199 per monthLoop11.com $350 per project

• Cost ($)

Advantages

• Cost ($)• Cost (Time)

Task In-Lab WebRequesting participants 20 minutes total;

explaining criteria to recruiter 30 minutes total;

specifying test groups and criteria

Screening participants 10 minutes per participant 0 minutes; done by TryMyUI

Scheduling participants 15 minutes per participant 0 minutes; study done at participant convenience

Preparing for interviews 10 minutes per participant 60 minutes total; setting up web survey and tasks

Total (for 45 interviews)

26.6 hours 1.5 hours

Data collection period 3 weeks; based on interviewer schedule

1 – 2 days;Depends on participant criteria

Advantages

• Cost ($)• Cost (Time)• Short data collection period• Participants that can be selected based on criteria• Videos can be shared• Participants are skilled/trained – At thinking aloud– At dealing with problems– At evaluating websites

• Can get replacements for unusable cases

Disadvantages

• No follow-up• Cannot correct navigation errors• Limited to 20 minutes*• Task completion time, success, and other quantitative

methods have to be captured manually*• No rating scales*• Panel participants may have selection biases that

make them different from a ‘typical’ user

* May be different on other testing services

Discussion

OpenHallway.com

UserTesting.com

Loop11.com

UTest.com

TryMyUI.com

• What testing service have you used?• What are the advantages/disadvantages of

that service?

Userzoom.com

OptimalWorkshop.com

Contact Information

Brandon KoppResearch Psychologist

Office of Survey Methods Researchwww.bls.gov/osmr

202-691-7514kopp.brandon@bls.gov