user research on a shoestringsusan teague-rector and erin white

vcu libraries, richmond, va

What we'll coverImproving user experience of your (web) product through research before and during your design and development process.

  Not spending a lot of money.

      Taking the scary out of usability.

What we'll coverBefore you begin:Evaluating use of what you have   

• Heatmapping• Analytics

As you work:Assessment of user experience  

• Virtual user testing • Guerrilla testing• Microfeedback

Some context

Sound familiar? Large university (32,000 students), small Libraries web team (3 folks), and many projects (~1,000,000). Your situation in a nutshell: • There's never enough time;• spending the little money you have is a hassle, but• you still want to make data-driven design decisions.

 Optional addition:• Your boss wants you to justify user research.

 

Justifying a little research

If your supervisors still need coaxing (or if you do), here's why any teeny little bit of user research is going to make your life easier: • Research can significantly decrease development and maint

enance time, which means cost savings.

• Google is right. Focus on the user and all else will follow. • Happy users = brand buy-in. When users like the library

more, everybody's happy.

Let's get in it

We're going to use a case study of the last VCU Libraries web redesign to demonstrate some of the tools we used to do user research.

The year: 2009. The goal: redesign.

• Heatmapping• Analytics

 Combined with • Existing feedback • Library literature

• Virtual usability testing

• Guerrilla tests

• Heatmapping• Microfeedback

originalhomepage

prototypes new homepage

The plan

Heatmapping

• What are you measuring? o Where are people clicking? What is getting attention?

• What tool to use?  o We used CrazyEgg and tracked clicks on the homepage

for a week in the middle of spring semester.• What does it track? 

o Clicks based on network and browser properties and return visits, among others. 

• How does it work? o Insert the javascript into the page you want to measure,

then let CrazyEgg do the work.

Heatmapping results

Heatmapping: pros and cons

Pros• Robust statistics with minimal effort, $$$• Fast way to evaluate components of a page • Reports are easily PDF-able and aren't overly technical, so

you can easily have a document to send up to managers.• Invisible to users.

 Cons• Clicks don't tell the whole story. We know where people

click, but we don't know if folks are finding what they want, or if the page makes any sense in the first place.

We've done post-launch heatmapping, too.

Analytics

• What are you measuring? o What pages get the most traffic?o What search terms are people using to get to your site?o What are people searching for on your site? 

• What tool to use?  o We used Google Urchin and an in-house search query

tracker. (Removing personally identifiable data like IP.)  o Google Analytics does this as well.

• How does it work? o Urchin analyzes server logs of page visits, user

characteristics, and referring sites.o In-house tool records search terms into a database and

displays a word cloud of frequent terms.

Analytics: Urchin

Analytics: Search terms

Analytics: pros 'n' cons

Pros• Search terms tell us what pages are hard to find, and

ways users think about our site. • Page visits tell us what's popular, and where people are

coming from to get there.• Analytics tracking is invisible to users.

 Cons• These tools can't track user navigation paths through our

site. • Again, clicks don't tell the whole story. Use data only tells us

so much.

And now...Usability research.

(I found this on the internet years ago, and if you can tell me what this is from or who the artist is, please e-mail me. It's driving me nuts. - Ed.)

User experience: what we're here foror, what is UX?

Morville, P. The User Experience Honeycomb

What smart people say about UX

Peter Morville gets us thinking about UX as a honeycomb of  elements that together can form a positive user experience.

We list several UX resources at the end of the presentation, too.

Usability and UX: What's the diff?

• Usability is a component of User Experience, the part of this equation that you have control of.

Demystifying usability

Usability research just asks, "Does this thing make sense to other humans?"   There is no perfect tool.

Usability engineering is a process, not an end result.  Usability research can be easy, fun, fast, and inexpensive. 

And anyone can do it.

Krug on how we should think about usability studies:  some > none

done > perfect

sooner > later

OCLC # 61895021

This book rules (and is a fast read)

Goals of usability testing

Jakon Nielsen: Usability 101 • Learnability: How easy is it for users to do basic tasks the

first time they see the site?• Efficiency: Once users have learned the design, how

quickly can they perform tasks? [in that session]• Memorability: When users return to the design after a

period of not using it, how easily can they reestablish proficiency?

• Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors? 

• Satisfaction: How pleasant is it to use the design?

So let's get crackin'

We tried to evaluate user experience through virtual usability testing with Usabilla, in-person guerrilla usability testing, and microfeedback forms.

All these methods are components of a UX toolkit. Individually, they are helpful, but together, they give a more complete picture.  There is no "right" answer, but with a combination of tools you can learn more than with one tool alone.

Virtual usability testing

• What are you measuring? The intangibles of user experience.o What doesn't make sense?o Where do you think you should click to perform X task?o What works?

• What tool to use?  o We used Usabilla. 5secondtest and a host of others

provide similar services.• How does it work? 

o Upload a screenshot of a web interface and create questions to ask about the interface. Questions can be task-based or feelings-based.

o Users visit the testing site on their own time and give feedback in the form of clicks and optional notes.

What do you like about this page?

What do you like about this page? (comments from users)

 Notes• nice white space around outside• nice centralized location for web 2.0• boxes arranged in tight enough formation that

they aren't just "a page of text"• dynamic (?) changing image centralized

What do you dislike about this page?

What do you dislike about this page? (comments from users)

  Notes • hours at top, directions bottom left, events bottom

right, services in the middle ... i'm lost• 'contact us' twice within inches of each other?• 'suggestions and feedback' (what's the

difference?) down here, 'ask us' up top• what could possibly be in 'about us?' it all seems

to be on the homepage ...• location of FOL thing near web 2.0 stuff is bad --

confuses def'n of "friend"• will this be consistent branding throughout?• not sure if people will understand library services

a-z w/o prompting

What do you dislike about this page?

  Notes • hours at top, directions bottom left, events bottom

right, services in the middle ... i'm lost• 'contact us' twice within inches of each other?• 'suggestions and feedback' (what's the

difference?) down here, 'ask us' up top• what could possibly be in 'about us?' it all seems

to be on the homepage ...• location of FOL thing near web 2.0 stuff is bad --

confuses def'n of "friend"• will this be consistent branding throughout?• not sure if people will understand library services

a-z w/o prompting

Changes we incorporated are highlighted.

Virtual usability testing pros and cons

Pros• Fast and easy for all involved.• Data on intangibles that we couldn't get through web stats. • Free-text suggestions as a catch-all for unforeseen issues. • Bonus: users could position free-text comments contextually

on the page, so we could easily figure out what the hell they were talking about when they said, "this thing here doesn't make any sense."

 Cons• It can be hard to find users who are invested enough to (a)

participate, and (b) give free-text responses.• Keep context in mind if you have a small, relatively

homogeneous sample (i.e. Library faculty).

Guerrilla testing

• What are you measuring? The intangibles of user experience: feelings, task completion, other thoughts about features and functionality.

• What tool do you use? Paper prototypes, basic web prototypes, or anything in between; and a little extraversion.

• How does it work?o Take advantage of the fact that your web users are the

same people who use your physical spaces. o Develop a short script, walk out into public spaces with

prototypes, seek out participants, and ask people questions.  

o Shut up and listen.

Example method

• Before you begin: develop a short script for use at the beginning and end of each interview, as well as a few questions for each user.

• Go out into the library and approach people who look like  they could spare a few minutes to talk. Try to target folks across populations - undergrads, grads, faculty.

• Shoot for a 5-minute interview. Keep it short, and try not to overwhelm.

• Don't worry about getting it perfect every time. • Write down everything you can - verbal and nonverbal

responses, gestures, facial expressions, etc. • Respect the user's time.• Listen.• Say thank you!

Example intro script

Hello, my name is _______ and I’m with VCU Libraries. We’ve been working on a redesign of the Libraries Web site and would love to get your feedback on the design. Would you be willing to participate in a short 5 minute survey?  We will not be collecting any personal information except your affiliation with the university.    This is also not a test of your abilities, but just a tool to help us understand how the site will work for our customers.   Do you have any questions before we begin?

• What is your affiliation with the university?• What single feature do like most about the web page?• What single feature do you like least about the web page?• How many Top Resources does the Libraries recommend?• In the redesign, how would you

o Find a book?o Find articles for your specific topic?o Place a hold or renew a book?o Order/find items that VCU Libraries does not own?o Contact a librarian?

• What single feature would you like to have available that you do not see included?

Example questions to ask:

Guerrilla testing pros and consPros• Most web developers have to seek out users; library users

are right here in our building.• It's an invaluable opportunity to talk to real users and learn

about problems you might not've even considered.• You will find usability issues FAST. Be prepared to be

flexible about addressing them.• You can repeat guerrilla tests as you iterate on designs.

 Cons• Users may not be invested enough to give in-depth

answers. Give small compensation if you can.• Be prepared to return to the drawing board (a good problem

to have!)• Be mindful of your biases when selecting participants.

Microfeedback

• What are you measuring? The intangibles of user experience during/after a new web product launch.

• What tool do you use? Small, 3-question quick feedback forms built in PHP. Some folks also use GetSatisfaction, UserVoice or Google Docs

• How does it work? o Create a prominent "Give Feedback" link to the survey, or

just embed the survey on your page.  Keep it small, fast and easy. Make targets big. Don't require fields. Let users decide what to tell you.

o Users submit quibbles/questions/comments.o Look at the comments, look for trends, and address

issues.

  

Love it

  

Like it

  

Indifferent

  

Confused

  

Hate it

Ease of use ° ° ° ° °Content ° ° ° ° °Layout ° ° ° ° °Overall ° ° ° ° °

The questions we asked:

Any other comments?

  

Love it

  

Like it

  

Indifferent

  

Confused

  

Hate it

Ease of use 93 65 20 21 38

Content 85 75 29 21 25

Layout 92 75 21 19 28

Overall 90 75 18 22 27

...and a host of free-text responses (n~=100).

Three months, 268 responses

Microfeedback pros and cons

Pros• Quickly pick up on unforeseen problems, broken links, and

usability issues.• Users can give as much or as little info as they want, fast.• Some users use microfeedback already and may be more

willing to use it (think Facebook's "like" feature). Cons• Some users thought this was a library help form. Be clear

about the form's scope when you post it.• The sample mostly includes lovers and haters, and not

many in between. Don't take mean responses (and there will be some) personally.

When doing user research, remember:  some > none

done > perfect

sooner > later

OCLC # 61895021

Remember this slide?

Thanks! and resources• User Experience Network • UXMatters • Garrett, J. J. The Elements of User Experience  • Krug, S. Don't Make Me Think• Morville, P. Ambient Findability• Nielsen, J. Designing Web Usability and UseIt.com• Designing Better Libraries • Adaptive Path • Quick and Dirty Remote User Testing

The authors

Susan Teague-RectorWeb Design Project LibrarianNCSU Libraries

Erin WhiteWeb Applications DeveloperVCU Libraries