Webinar: How to Conduct Unmoderated Remote Usability Testing

Webinar:How to Conduct Unmoderated Remote Usability Testing

Intros

Alfonso de la NuezCo-Founder & Co-CEO

UserZoom

Nate ColkerUX Researcher

UserZoom

John RomadkaUX ResearcherOptum Insight

(Division of UnitedHealth Group)

OverviewPart 1 (20 min)

Introduction to unmoderated remote usability testing A real life case study featuring a company in the insurance industry

Part 2 (30 min)

How to plan for an unmoderated remote usability study How to design a study What metrics you need to collect How to recruit participants How to analyze the data

Q&A

User testing, feedback & analytics tools mania!

Let’s try to make some sense out of this…

Many ways of organizing remote testing tools: Moderated vs. Unmoderated Qualitative vs. Quantitative-based User engaging vs. Non-user engaging (analytics) Usability focused vs. ‘quick user feedback’ (VOC) Video vs. Stats/metrics Etc.

Remote, unmoderated - A growing research method

Why this testing tool fever?

Consumers more empowered than ever, thx in part to Social Media

Ecommerce maturing, more competition

Web analytics, usability test in the lab, mkt research… no longer enough or are very costly

Agile dev is all about speed, iteration

1. Designing & Testing UX quickly & cost-effectively is KEY

2. People want better!

Why Should You Care about Remote Testing?

Cost-effective, save time and money on research You can quantify usability, obtain statistically significant UX

data Run competitive UX Benchmarking Iterative testing, validate design/prototypes Test with users from different geographic locations Objectivity + accountability Improves communication Mitigate risks

What can be tested with URUT?

Static mockups Wireframes Prototypes Multimedia Live websites Intranets

…any web-based UI

…anything you can host on a server and access through a web browser

Pros & Cons

Pros: Cost-effectiveness Quantify UX Geographic representation Honest feedback Combination of results and metrics

Cons: Obviously… no face to face

Case Study:Remote UX Studies - Tools & Methods

John Romadka, UX ResearcherOptumInsight (Division of UnitedHealth Group)

ChallengesNew project We’d depleted our pool of internal (agent) participants locallyNeeded to get a broader perspective from agents across countryNetwork security issuesLimited number of dedicated UX Researchers

ChallengesCouldn’t convince business partners to test iterativelyLab studies were costly and often resulted in large margin of error

90% confidence interval

Making the CaseMade the unmoderated testing business case

More cost efficient • Unmoderated testing can cost 50% less

than traditional moderated testing• Primarily due to reduction in effort hours spent in lab

Timeline• Reduced the 3 week lead time for recruiting

to less than 1 week.

Making the CaseValidation Testing

Retest within days, instead of weekModerated test: 180 hours vs. Unmoderated test: 40 hours Remote Unmoderated Testing performed for ~20% the cost of benchmark Moderated test

Test 1- (Moderated) = $10,000Test 2- (Unmoderated) = $2000)

Improved Confidence Intervals

ResultsSuccessful company-wide adoption of UserZoom

Obtained a pilot group of “Field” participantsBroad range of demographics, locations

Bi-weekly expectation of participating in whatever UX research was available and ready to study• Required less context, setup

ResultsUserZoom allowed us to quickly setup/run/analyze

Online SurveysTask-based studies Card Sort exercisesTree Testing studies

Task-based studies:Moderated Remote

Unmoderated Remote

Moderated Lab

ResultsTask Based Studies - Remote Moderated: combine 3 tools

UserZoom (metrics)Lead users through studyCentral point for metricsQuick and easy to re-test

Webex (screen sharing, audio)Deployed company-wideGiving participant control of web app on dev

Morae (note taking, recording)

ResultsTask Based Studies - Combined Unmoderated/Moderated

Users were onsite occasionallyValidate our unmoderated results

UserZoom: Used same setup for bothConsistent test method and processFreed up the Facilitator to observe, take notes.

Knowing UserZoom was gathering metrics

Thank You!

@jaromadGplus.to/jromadka

How to Conduct Unmoderated Remote Usability Testing

• How to plan for an unmoderated remote usability study

• What metrics you need to collect

• How to design a study

• How to recruit participants

• How to analyze the data

Planning Design Recruitment Analysis

What are the goals of the study?

A Few Examples….

Evaluate and Quantify the User Experience

Comparing Multiple Designs (A/B)

Competitive Benchmark

Identify Visitors’ Core Tasks

Redesign Navigation Structure

Improve the Navigation Experience



A Few Examples….

Remote Task-Based

True Intent

Card Sort / Tree Test








Method

A Hulu Demo Study


What Metrics do you Need to Collect?

Jeff Sauro, Measuringusability.com


Create A Study Script

Recruitment AnalysisPlanning Design

• Study Goals & Participant Profile

• Welcome Page

• Initial Questions

• Task 1

• Post-Task Questions

• Task 2

• Post-Task Questions

• Final Questionnaire

Welcome Page Design


Set Expectations for the participants

• Introduce study

• Time estimate and incentive

• Keep short and to the point

Purpose of the study

Time estimate

Incentive

Technical requiremen

tsInstructions

Example Welcome Page


Initial Questions - Tips


You may want to collect what type of experience users have with the product.You can use branching logic to give certain tasks to specific participants based on their expertise

Avoid asking hypothetical questions. Participants are not good at knowing how they’ll behave in a certain situation.

“Would you book flights on this website if we added this option?” is a bad example

Be specific. “Daily” and “Weekly” is more specific than “Always” and “Sometimes”.

Avoid “double questions” that have two questions in one.

“Please indicate how easy and fast it was for you to complete the task?”

Avoid asking yes/no questions. It can lead participants into an extreme statement. “Do you use this website all the time?” is not a good example.

Don’t reveal too much information to the participant upfront Brand recognition can have a large impact on people’s reactions to products.

Initial Question Examples


Initial Question – Logic & Conditions


• Skip participants to a specific task or question

• Hide or show questions based on answers (example above)

• Disqualify participants

Selecting the Right Tasks


Balance of User Data and Business Goals

• What are users coming to your site to do (core tasks) ?

• Are there any tasks that are required to use your site?

• What drives revenue on your site?

Selecting the Right Tasks


Balance of User Data and Business Goals

• What are users coming to your site to do (core tasks) ?

- Find and watch a TV show

• Are there any tasks that are required to use your site?

- Create an account?

• What drives revenue on your site?

- Sign up for a premium account

Task Construction - Tips


Tasks should be natural and easy to understand, but not easy to guess

Randomize task order to minimize the learning effect, if tasks are not dependent on each other.

Create tasks that are easy to validate (you need to be able to confirm that the participant actually found an answer).

Tasks should be worded in a way that doesn’t lead the participant

When using Validation word tasks so that there can be a discrete answer (price, date etc.)

Include “none of the above” option (it will discourage participants from using deduction to figure out the answer).

Building Multiple Tasks - Randomizing Order


Screener Pre-task questions

Hulu Task 1

Hulu Task 2

Control for learning effects

• Participants become more proficient as they use the site

There are exceptions, sometimes tasks must be completed in

order

Final QuestionsRandomized

Task Validation (3 options)


• Validate by question

• “Quiz question” used to verify task completion

• Validate by URL

• Successful if participant reach a specific URL

• Think carefully about task wording and defining success • “Find episode 5 season 5 of Family Guy” (Is someone successful if they get to search

results page vs. the specific episode page)

• Self-reported

Task Description Example


Post Task Questions


Self-Reported Data

• Are user satisfied with the experience?

• Do users believe they completed the task successfully?

• Why wasn’t a participant successful ?

TaskSuccess Button Validation

Abandon Button Abandon Questions

Error Questions

Success Questions

Post Task Question Examples


Validation Question

Error Question

Success

Final Questionnaire - Tips


Overall, how easy or difficult was it to use this website?

How likely is it that you would recommend this website to a friend or colleague?

After your experience with the website today, which of the following words do you associate with the brand?

When using Validation word tasks so that there can be a discrete answer (price, date etc.)

OVERALL RATING SCALES

NET PROMOTER SCORE

CHANGE IN BRAND PERCEPTION OPEN-ENDED QUESTIONS

Final Questionnaire Example Questions


What about recruiting participants?

AnalysisPlanning Design Recruitment

Define a screener

Hulu Demo Study

• Representative users

• Demographics (gender, age etc.)

• Control for experience

• Larger sample size -> more precise predictions

• Set profiles and define quota limits

• Set quality controls

Quality Control Settings


Quality Controls

• Be careful not to exclude fast users

• Cheaters often take the most efficient path (look out for “0

clicks”)


3 choices to recruit participants:1. Panel vendor (SSI, Research Now,

etc)2. Private Mailing List3. Intercept visitors


Hulu Demo Walkthrough


Analyzing the Results


1. Task Success

2. Efficiency (time, clicks)

3. Self-Reported Satisfaction

4. Open prompts

5. Behavioral Data

Analyzing the Results – Clean Data


Review your data for outliers / “bad data”

• Very High / Very Low task times

• 0 Clicks

• Nonsense Answers

Make assumptions about user population (Confidence Intervals)

Hulu baselines can be compared with data from future study (Significance

Testing)

Analyzing Metrics – Beyond the Tool


Jeff Sauro, StatUsabilityPak

Analyzing Metrics – Open Ended


Use Text Clouds & Search for Patterns

Use Comment Coding for more Detailed

Analysis

Analyzing Metrics – Behavioral


What strategies are being used (how successful are

they?)

What are “Non-Success” participants doing? (filter

data)

Thanks so much for your time!Check out our upcoming news and events at www.userzoom.com

Sunnyvale (USA)

440 N. Wolfe Rd. Sunnyvale,CA 94085 Phone: +1 (408) 524 7445 Contact: Alfonso de la Nuez

Barcelona (Spain)

Av. Diagonal 419 3º 2ª 08008 Barcelona Phone: +34 93 414 7554 Fax: +34 93 209 8380 Contact: Xavier Mestres

Cheshire (UK)

Booths Park. Chelford Road, Knutsford, Cheshire WA16 8GSPhone: + 44 (0)1565 759890Cell: + 44 (0)7900 472 920Contact: Arthur Moan

Webinar: How to Conduct Unmoderated Remote Usability Testing

Devices & Hardware

Transcript of Webinar: How to Conduct Unmoderated Remote Usability Testing