Post on 28-Jan-2015
description
Research Traps: 7 ways of thinking that keep you from doing great customer research
Wendy CastlemanPrincipal User Research Scientist
To be presented at the UPA Conference in Portland, OR in June 2009
Who am I?
ExperienceResearch
RESEARCH TRAPS
Mental Shortcuts
Mental shortcuts help us
• Make quicker decisions to take action faster…
Those shortcuts become research traps
Awareness
7
7Habit
FalseConsensus
Congruency
ConfirmationAvailability
Recency
Illusory Correlation
Meet Elsie
What do you think happened?
I lived hereI moved here
maps.google.com
Habit
We tend to do thingsthe way we usually do
New project?
Sure, I can run
a usability test!
Habit Trap
Can lead us to do the wrong research
Your favorite research method may not be the best way to learn what you need to know
The best way
to find that out
would be a site visit…
How to Avoid the Habit Trap
•Look at every project as unique•Consider what you need to learn • Identify the best method
• If Yes = guess most people would agree
• If No = guess most people wouldn’t agree
Eat
at Joe’s
False Consensus
We tend to attribute our beliefs,thoughts and behaviors to others
Hi guys!
Wanna be in a study?
False Consensus Trap
Test with the wrong participants
Other people may not think like you…
Junior League of Palo Alto
Avoiding the False Consensus Trap
•Focus on the customer–spend time watching and talking
•Test with people who aren’t like you
What is the rule?
2, 4, 6, 8, ___Hypotheses: Each value must be 2 higher than the one before.
How do you test this hypothesis?
Actual Rule: Each value must be any number bigger than the one before.
Congruence Bias
We jump to conclusions by only looking at one approach
Try out our new idea for an iPhone application!
Congruence Trap
Only trying one solution may miss a better one
insufficiently inform the design
Try out each of these iPhone applications!
Avoiding the Congruence Trap
•Test several different solutions•Test out what shouldn’t work
Which card do you turn over?
Hypothesis:
The back of every “6” is an “L”.
T 6 L 4
Did you say “T”?
Confirmation Error Bias
We have a tendency to search for data to confirm expectations
As I suspected!
25% of users
can’t do that task!
Confirmation Trap
Get an incomplete picture of the data
Seeking to prove our ideas can lead to missed surprises
Quicken used by small business
Avoiding the Confirmation Trap
•Look for surprises, instead of what you expect•Test out what shouldn’t work•Consider independent evaluation
In American English are there more:
Words that begin with the letter “K”?Or
Words where “K” is the third letter?
There are 3x as many words with “K” in the 3rd position
Availability Heuristic
We have a tendency to put too much weight
on what comes easily to mind
Availability Trap in Research
…a lot of people had trouble with that
task…
Availability Trap
Draw inaccurate conclusions
What comes to mind easiest may not be the most important or most frequent finding.
Hmm… I didn’t realize that
happened so often…
Avoiding the Availability Trap
•Gather key usability metrics –(task success, specific error counts, time)
•Don’t rely on your memory•Look at all of the data –Encourage others to do the same
Last 3 moviesPrior 3 movies
Recency Bias
We tend to put too much weight on what we saw most recently
Recency Bias in Research
Click “Continue”
Participant 5
Click “Continue”
Participant 4
Click “Done”
Participant 3
Click “Done”
Participant 1
Click “Cancel”
Participant 2
Most people clicked “Continue”
Recency Trap
What you saw most recently may not be the most important or most frequent finding.
Draw inaccurate conclusions
Hmm… I didn’t realize that
happened so often…
How to Avoid the Recency Trap
•Gather key usability metrics –(task success, specific error counts, time)
•Don’t rely on your memory•Look at all of the data
Hap
pin
ess
Weight
Illusory Correlation
The tendency to find patterns where none exist
This is the third guy
who uses a laptop
in his living room.
Maybe all men
use laptops
in their living rooms
Illusory Correlation Trap
Things that co-occur may not be related.
Draw inaccurate conclusions
That’s the fourth man
who has bought this version.
I need to find out how
many men buy this…
Avoiding the Illusory Correlation Trap
•Recognize the limitations of your research methods
•Verify magnitude estimations and correlations with large-scale quantitative studies
7Habit
FalseConsensus
Congruency
ConfirmationAvailability
Recency
Illusory Correlation
Ways to avoid the traps…
Planning1. Look at every project as unique2. Consider what you need to learn 3. Identify the best method4. Test with people who aren’t like you5. Test several different solutions6. Test out what shouldn’t work
Ways to avoid the traps…
Conducting1. Look for surprises, instead of what you
expect2. Gather key usability metrics 3. Consider independent evaluation
Ways to avoid the traps…
Analyzing1. Don’t rely on your memory2. Look at all of the data3. Recognize the limitations of your research
methods4. Verify magnitude estimations and
correlations with large-scale quantitative studies
QUESTIONS?Email me
http://Deepunderstandings.blogspot.com
Wendy_castleman@intuit.com