True Intent: The Best Online Benchmark You've Never Measured
-
Upload
uxpa-international -
Category
Spiritual
-
view
1.869 -
download
0
description
Transcript of True Intent: The Best Online Benchmark You've Never Measured
Ted BorenUser Research Lead
The Church of Jesus Christof Latter-day Saints
THE BEST ONLINE BENCHMARK YOU’VE NEVER MEASURED
TRUE INTENT
Is our site successful?
GAPS IN BENCHMARKING
What do visitors want to do?
(And can they do it?)
Where dothey go?
(How longdoes it take?)
What do theythink of theirexperience?
(Who arethey anyway?)
Whatyou want
to know aboutsite visitors’Experience
…
RemoteUsability Testing
WebAnalytics
OnlineSurveys
TrueIntent
Whatyou want
to know aboutsite visitors’Experience
…
“True Intent” in a Nutshell1. Intercept live visitors.2. Ask them why they came to your site.
3. Ask them to go do that.4. Track where they go and how long it
takes.5. Let them tell you when they succeed—
or give up.
6. Ask satisfaction and profile questions.
True Intent Studies are “Time-Aware”“Remote research opens the door to conducting research that also happens at the moment in people’s real lives when they’re performing a task of interest.”
“Time-awareness in research makes all the difference in user motivation: it means that users are personally invested in what they’re doing because they’re doing it for their own reasons, not because you’re directing them to; they would have done it whether or not they were in your study.”
~ Nate Bolt and Tony Tulathimutte, “Remote Research,”
page 10 (italics added).
CASE STUDY: True Intent Study for LDS.org*
* The flagship website for the Church of Jesus Christ of Latter-day Saints, my employer
Give me the details on thisTrue Intent thing!
“WHEN, HOW, WHO?”
WHEN in the development cycle do I do this?During planning…
… or after deployment.
BEWARE: Variation by hour, day, or special event
Personal study
Find something for my family
Prepare a lesson
Prepare a talk
Search Intents on LDS.org
Personal study
Find something for my family
Prepare a lesson
Prepare a talk
BEWARE: Variation by hour, day, or special event
Search Intents on LDS.org
TYPICAL STREAM: A Whole Week? Day? Month?
Search Intents Declining on the Weekend (spiking on Monday)
Personal study
Find something for my family
Search Intents Spiking on the Weekend
Prepare a lesson
Prepare a talk
So if in Doubt—Slow It Down and Spread It Out!• If you invite 100% of your visitors, you may finish too quickly, especially with:• Heavy traffic• Enthusiastic audience
So:• Meter the rate (preferred)*• Or turn your survey off and on
WHO do I invite? (And how?)•Email from a pool?•Link on the page?•Pop-up on the page?
Sample Popup Invitation
Sample Instructions
Sample First Question
True Intent’s Golden Question:
Why did you cometo this website
today?
Please be as specific as you can.
Prepare a lesson.
80% of your responses may be as exciting (and detailed) as this one.
But the other 10%...
My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.
~ LDS.org visitor
My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.
My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.
My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.
its my homepage. helps remind me who i am everytime i get on the computer.
~ LDS.org visitor
So when asking the Golden Question…
Beware of Qualifiers!
And Seize Today!
The One (and only) Task:“Do What You Came To Do.”
Visitors make their way through your site…
?
… but sometimes they may be hard to track.
?
The tool continues tracking time and path…
… until they say they are done.
Primary Post-Test QuestionsGoals
Satisfaction & Relevancy
Problems, Anything Else?
Secondary Post-Test Questions•Key pivots for slicing and dicing•Demographics / profile•System Usability Scale (SUS) or similar instruments…only if external comparisons are important and feasible
•Be thinking ahead to future comparisons…how will you know you’ve improved?•What data will speak to your stakeholders?•But KEEP it SHORT!
Q: How long do you run this thing?A: “It Depends.”•Site volume• Percentage invited•Survey length•Audience engagement… and incentives*
• I’ve finished in a day or two, or in 4-6 weeks.
• Shoot for at least 300-400 responses:+ More variety of audience and intents
+ More subgroups to slice and dice
= More responses needed
A FEW QUESTIONS?
300-400 responses later…
“ANALYZE THIS!”
Basic Quantitative StatsSuccess Rate
Average Satisfaction Ratings
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%Completed task
Completion Status
I found what I wanted.
It was easy to find what I wanted.
Within the next few weeks, I'll probably study, share, or use the information I found today.
1 2 3 4 5 6 7
StronglyAgree
StronglyDisagree
Breakouts: Stated Goals, by Age Range
• Younger LDS.org visitors more likely looking for personal study or inspiration…
• Older visitors come for these reasons too, but also more for utilitarian purposes.
Personal Study
Find something inspirational
Find something for my calling
Use the calendar, directory, maps, or leader tools
55 and older
35 to 54
18 to 34
Benchmarks: LDS.org Search SuccessVersion 2 was better overall:• Especially for
preparing a lesson…
• But not so much for finding something inspirational…
Prep
are
a less
on
Find
som
ething
insp
iratio
nal
Version 1 Version 2
ANALYZING QUALITATIVE DATA:The Affinity Diagram• A thematic, hierarchical view of qualitative data points.• A way to see what issues arise out of the data,instead of imposing a pre-determined structure.
Sample Affinity Diagram Excerpt:1. I visit LDS.org often for inspiration and information
1.1. I want to be inspired and strengthened
1.1.1. I am looking for something to help me feel the Spirit
1.1.2. I want quick inspiration during a break in my day
1.1.3. I start on LDS.org to keep “spiritually minded”
This is the data we start
with...
These are categories we generate.
Intent: “I try to read a quote or watch a Mormon Message every time I get on the internet. It helps me stay spiritually minded.” (Female, 35 yrs)
Intent: “I come every day as I start my browser, to keep me focused.” (Male, 62 yrs)
Intent: “LDS.org is my homepage. It helps me remember who I am and what I stand for before perusing the internet.” (Female, 17 yrs)
Intent: “Its my homepage. helps remind me who i am everytime i get on the computer” (Male, 25 yrs)
Preparing Data for the Affinity Diagram•Export the data to a spreadsheet•Create one row per comment*, including all data needed for context•Use “mail-merge” functionality to pull quotes and key contextual data (age, experience, success ratings, etc.) into a document formatted for printing the notes.•And then…
Meet Your New Best Friend…
•Post-It Notes for Printers•Print a few merged notes onto regular paper to check alignment•Print a few at a time on manual feed to avoid jams
Organizing over800 comments
Naming and grouping169 categories(11 at the top level)
Conduct an Affinity Diagram Workshop
Finished diagram, with three levels of headings.
Record the DiagramHeadings OnlyRecord only the headings in a document or spreadsheet
More CompleteAssociate each data point with its headings in a spreadsheet
Formal ReportRe-merge the headings and data back into a formatted document, with an introduction, findings, etc.
REPORTBackgroundKey Findings
A FEW QUESTIONS?
Analysis Steps LDS.org Example
1. Filter to those who a failed a specific task. *
Looking for a location
2. Review all the data you have for each instance:• Stated intent
• Path taken, search terms, etc.
• Biggest problem• Other survey questions
that might be relevant
Trying to find the DC Temple
Home > Maps < Home > Directory < Home > Temples > Find a Temple > Washington DC Temple
Didn’t see Temples under Families
Identify patterns and themes Temples not seen under “Families”
Suggest solutions, or follow-up questions
Broaden the heading.Conduct some tree testing.
Failure Analysis
Failure Cases for LDS.orgAlmost all problem cases fell into one or more of the following categories:•Difficulty finding things in menus•Problems with lack of quotation marks in Search•Content lacking or spread across multiple sites
Mitigating ChallengesIt can be difficult to get enough context to conduct an effective failure analysis.This works best when you have:• Additional information derived from the URLs captured (e.g. search terms or other parameters)• Live pilot sessions to provide context for common experiences.• “Biggest problem” survey data.
Be ready to accept that some paths will still be inexplicable.
What Next?•Conduct interviews or tests on key areas.•Adjust or create audience segments. Enrich personas.•Prioritize top intents for future development(or conduct a follow up survey to quantify themes from the affinity diagram).•Fix common problems from failure analysis.
•Re-test after you’ve had new changes deployed for a while; compare quantitative results.
Archive It!
But don’t just forget about it.
True Intent data has a good shelf-life…
How Long Does This Take?• For 300-400 intents:• Creating the online study, collecting data, running quantitative
stats:40-60 hours for the facilitator*• Creating the affinity diagram:
20-40 hours for the facilitator*• Failure analysis:
20-60 hours for one analyst**
• Total: 80-160 hours of your time, plus team time• Can be less with shortcuts• Should be less the second time around
But is it worth it?
Some unsolicited quotesfrom team members on
three different projects…
“Thanks for leading this exercise. I will admit I was a bit skeptical going into these meetings (who isn’t skeptical of meetings that take you
away from already-pressing deadlines?). But I’ve been really pleased with the process and
results, and I think they will be of great help in determining our future strategy.”
~ Member of Team 1
“I mentioned this study in two different high-level meetings this week, and
there is a great deal of interest and enthusiasm about the way the study
was done and in seeing the results.”
~Member of Team 2
“Thanks a ton for your work. As I’ve said before, the results of the study
will help inform our policies & site development going forward in big ways.
We’ve learned a lot – I’ll share appropriate info with [other key stakeholders].”
~Member of Team 3
No time for all of this?
Here are some ways to trimtime and budget.
SHORTCUTS
Shortcut #1: TRIM SCOPESkip 1 or 2 of the 3 main activities:•quantitative benchmark•affinity diagram•failure analysis*
Or collect the data for all 3, but wait on the analysis for 1 or more.
Shortcut #2:USE WORD CLOUDS•One cloud per open-ended question(or per set of closely related questions)•Dump all comments into a single text file•Standardize closely related word forms
“Please tell us why you came to this website today.”
“What was the biggest problem you encountered in your session today?”
“Anything else you'd like to tell us about your experience today?”
Conduct a “True Intent” Study1. Intercept live visitors.2. Ask them why they came to your site.
3. Ask them to go do that.4. Track where they go and how long it
takes.5. Let them tell you when they succeed—
or give up.
6. Ask satisfaction and profile questions.
Shortcut #3:Conduct a “True Intent” Study SURVEY1. Intercept live visitors.2. Ask them why they came to your site.
3. Ask them to go do that.4. Track where they go and how long it
takes.5. Let them tell you when they succeed—
or give up.
6. Ask satisfaction and profile questions.
FINAL THOUGHTS
Evaluating success on an actual, self-defined
task
Largenumbers
of paths andtimes on task
Feedback& profile data (qualitative &quantitative)
TrueIntent
We could call this:A single-task
remoteunmoderated
interview-guidedusability test
with integrated analyticsand survey questions.
But it’s really all about:
True TasksTrue Success
TruePaths
True ProfilesTrue Satisfaction
TrueIntent
FINAL QUESTIONS?
References & LinksOfficial Website of the Church of Jesus Christ of Latter-day Saints:• http://lds.org
Nate Bolt and Tony Tulathimutte, “Remote Research”• Book: http://amzn.to/18nEWtb
• Supplementary Site:http://remoteresear.ch
Jared Spool, “The KJ Technique”:• http
://www.uie.com/articles/kj_technique/
Caroline Jarrett, “Surveys That Work”:• http://rosenfeldmedia.com/books/s
urvey-design/blog/do_incentives_help_to_improve/
Creative Commons Licenses:
• Attribution, Non-commercial: http://creativecommons.org/licenses/by-nc/2.0/
• Attribution, Non-commercial,No Derivatives: http://creativecommons.org/licenses/by-nc-nd/2.0/