Fast, Cheap, and Actionable

Creating an Affordable User Research Program

Michael PowersDirector of Web Services

• museyroom.com

• twitter.com/mjpowers

• www.slideshare.net/michaelpowers

• www.linkedin.com/in/mjpowers

On the Web

What kinds ofuser research

have you done?

It all started with ausability test

Today• Getting Permission for Research

• A Process for User Research

• Inventory of Actionable User Research You Can Do Fast and Cheap

User Research Study↓

User Research Program

GettingPermission0

Don’t Ask

“We don’t want “a research“study.”

“A research study?

“Look, just make a red box on the home page, and put this text in there.

“I just need to make sure students don’t miss it.”

Nobody wants aresearch study.

Nobody wants a website.

What We Really Want• Applicants

• Successful Students

• Academic Research

• Donations

Create a Research Culture• Do a usability test and invite

stakeholders to watch

• Make research part of every project

• Apply previous research to new projects

• Show results early and often

Research Process1. Frame a Good

Question

2. Know What You Already Know

3. Choose Study Type

4. Find Participants

5. Run Study and Analyze Results

6. Take Action

7. Measure and Repeat

Frame aGood Question

1

Good questions are related to your goals

“Students can’t find stuff on our website.”

Focusing a QuestionHow can we decrease attrition?

• Can students find information about support services?

• Can students find out what’s happening on campus?

“The admissions website needs work.”

Focusing a QuestionHow can we get more/more qualified applicants?

1. Can prospectives find our list of majors? Our tuition information?

2. Can prospectives register for a visit?

3. Can prospectives successfully apply on line?

Goal-related questions lead to faster, cheaper, more actionable user research

Know What You Already Know

2

What Do You Already Know?• Best Practices

• Published Studies

• Data about Your Institution and Students

• Server Logs

• Analytics

• Past Studies

Get OrganizedQuantitative Data

KPI Spreadsheet

Qualitative Data

User Personas

KPI =KeyPerformanceIndicator

KPIs measure progress toward your goals.

KPI Spreadsheet• Focus on the KPIs you've

chosen

• Give specific reports to stakeholders who request them

• Track over time without running multiple tools

• Include anything numeric: survey results, 4Q survey, analytics, server logs, etc.

• Create specific analyses for specific projects

KPI SpreadsheetAvailable on museyroom.com

User Personas

Include ScenariosMotivation Scenario

Site Features Used Behaviors

Find the school with the right pre-Law program for me

Checks to see if her major is available

•Site Search

•Undergrad Admissions Website

•Academics Link

•Looks for “Pre-Law” major, does not find it.

User Personas• Fiction—but a useful

fiction

• Puts what you've learned into a format you can share with your team and clients

• If it is wrong, team or clients will let you know

• Helps bring new team members up to speed

Use these two tools to track your progress

over time.

Choose Study Type

3

If you’ve never done one, do a usability test first.

FocusGroups

Some Types of User Research• Usability Test

• Surveys

• Pop-up Survey

• KJ Session

• Card Sort

• Navigation Test

• Layout Test

• A/B Test

Some Types of User Research• Usability Test

• Surveys

• Pop-up Survey

• KJ Session

• Card Sort

• Navigation Test

• Layout Test

• A/B Test

Usability Test

Photo source: http://www.flickr.com/photos/l-i-n-k/3654390818/

Analytics

Survey

Pop-Up Survey

Pop-up SurveyWhat You Do: Ask site visitors questions about their

experience on a particular page or the whole website.

Good For: Capturing user intent; settling arguments about what belongs where

When To Use: When you have a question to answer, or all the time

Participants Needed: Calculate your sample size:http://www.surveysystem.com/sscalc.htm

Pop-up SurveyTools Available: •4Q Survey

•iPerceptions•CrowdScience•Roll your own

Cost: •$0–$10,000+

4Q Survey Questions1. Based on today's visit, how would you rate your site

experience overall?

2. Which of the following best describes the primary purpose of your visit?

3. Were you able to complete the purpose of your visit today?

4. Comment

• If yes: What do you value most about the website?

• If no: Please tell us why you were not able to fully complete the purpose of your visit today

“I am always able to find what I am looking for. The website is really easy to use.”

“I can never easily find what I'm looking for, if I can find it at all.”

Coding Comments• Visual Design

Positive

• Visual Design Negative

• Usability Positive

• Usability Negative

• Navigation Positive

• Navigation Negative

• Content Quality Positive

• Content Quality Negative

!" #!" $!" %!" &!" '!!" '#!" '$!" '%!" '&!"

()*+,)-./01+/2)3+4+56"

7)8"9./5:/5";"(::2"

<=:>+?>"=),:08+5:"

;@A"B:38+5:C"/./D9E<"

<+5:"<:)F>G"

9./5:/5"HI)4+56"

1+/2"A:.=4:"

@8)34:"

J+8I)4"K:8+,/""

(:L8")/2"M*:/58"

NF,)/+O)-./"

!"#$%&'()'*(##%+,-'.%+/(+0+1'230-'2(405'

2(405-'

637,'809':-%&-'*(##%+,';+<'

(:,)-*:"

A.8+-*:"

!"# $!"# %!"# &!"# '!"# (!"# )!"# *!"# +!"# ,!"# $!!"#

-./01.23456047.8090:;#

<.=#>34:?4:#@#-??7#

AB?C0DC#B.1?5=0:?#

@EF#G?8=0:?H#434I>JA#

A0:?#A?.KCL#

>34:?4:#MN.90:;#

6047#F?3B9?#

E=.89?#

O0=N.9#P?=014##

-?Q=#.47#R/?4:=#

SK1.40T.234#

!"#$"%#&'()*+,*-.()*"/#00*12&"

3#4'5&"

67'57"3#4'5&"8'9":&*;&"/#00*12"<#&2"%#&'()*=>+,*-.()*=>"?1@"

-?1.2/?#

F3=02/?#

Action• Changed our yearly goals to spend more time

on these issues

Card Sort

Photo Source: http://www.flickr.com/photos/rosenfeldmedia/3343501403/

Card SortWhat You Do: Give participants cards with website

topics on them, then ask them to put them into piles that make sense to them

Good For: Understanding the mental models of your users

When To Use: When creating or improving your information architecture/navigation

Participants Needed: 15 face to face50 online

Card SortTools Available: •Index cards

•OptimalSort (remote)•WebSort (remote)•xSort

Cost: $40-$600

Open• Users stack cards

then

• Give each pile a name

Closed• Give users header

cards• They assign cards to

the header cards

Where do we put counseling?

Action • Complete restructuring of the Student Life

section of our website

• Move counseling websites out of “Health” and into “Support”

One Subsite,Before and After

Change between Comparison Periods

Compared toWhole Site

Pageviews 76% 140%

Unique Pageviews 81% 142%

Bounce Rate -21% 27%

KJ Session

Photo Source: http://www.flickr.com/photos/drewm/370231088/

Navigation Test

Navigation TestWhat You Do: Give participants a proposed site

structure/navigation and let users drill down to find different things.

Good For: Verifying an IA in a more realistic way (not everything is visible); finding problem areas

When To Use: To verify a new information architecture

Participants Needed: 50

Navigation TestTools Available: •Treejack

Cost: $150–$700

Action• Rearranged Admissions Site navigation to

• Combine repeated material

• Put “Costs and Tuition” higher in the site tree

Layout Test

Layout TestWhat You Do: Participants respond to a static image

of your site

Good For: Behavior, Remote, Automated, Quant.

When To Use: Before building out a new design

Participants Needed: About 50 per test

Layout TestTools Available: •Paper

•PowerPoint•Five-Second Usability Test•Chalkmark•Usabila

Cost: $40–$600

Layout TestPositive Negative•Get reactions to real

designs without building out a whole site

•No interactivity

•Users can be confused when doing this remotely

A/B Test

Find Participants

4

The Right Number of Participants• Different methods have different requirements

• Plan to recruit more than you need

• But don’t test more than you need

The Right Kind of Participants• Steve Krug: “Recruit loosely and grade on a

curve”

• Good advice, but when possible try to recruit participants that are more like your users

• For instance: faculty and students see your website very differently

Finding Participants• People you know

• Students, faculty, and staff at your school

• Local high schools (talk to a guidance counselor)

• On your website (link on homepage or pop-up)

• Recruitment tools like Ethnio

Photo source: http://en.wikipedia.org/wiki/File:Ethnio_recruit_ad.png

Offer a prize

Run Study and Analyze Results

5

Keep track of results in KPI spreadsheet and personas

Take Action6

Share results and proposed solutions with stakeholders

Measure and Repeat

7

Share results

Challenge: Do some sort of user research each month

Research Process1. Frame a Good

Question

2. Know What You Already Know

3. Choose Study Type

4. Find Participants

5. Run Study and Analyze Results

6. Take Action

7. Measure and Repeat

More Informationmuseyroom.com

twitter.com/mjpowers

www.slideshare.net/michaelpowers