Beyond Measure
@mulegirl
DEEP THOUGHT,
TELL US THE ANSWER…
OK
7,500,000 YEARS LATER…
42!!#!?&wtf!
¯\_(ツ)_/¯
IT WOULD HAVE BEEN SIMPLER TO
KNOW THE ACTUAL QUESTION.
ONLY WHEN YOU KNOW THE
QUESTION, WILL YOU KNOW WHAT THE ANSWER MEANS.
I am a rational biped. All rational bipeds rule.
So, I rule.
We have to work together to make better decisions.
Quantitative approaches feel
better, but are often actually worse.
We are imperfect humans
designing things for other imperfect humans.
Flickr/fungleo
Kirk Siang/Flickr
Flickr/eskimo_jo
[dramatization]
[a dramatization]
The Nourishing Home / Kelly Smith
luckyironfish.com
oo
O
O
O
OHO
HO
OH
OHH2C
H2CCH2
CH2CH2C
C
CH2 C
C
:N
N:
Human problems require illogical
approaches.
Stories have
power
Flickr/ Billy Wilson
Only 12 people?!?
2,500,000 000,000, 000,000 bytes
01101000011001010110110001110000001000000110110101100101
010010010010000001100001011011010010000001100010011001010110100101101110011001110010000001110111011000010111010001100011011010000110010101100100
01101000011001010110110001110000001000000110110101100101
01101000011001010110110001110000001000000110110101100101
010010010010000001100001011011010010000001100010011001010110100101101110011001110010000001110111011000010111010001100011011010000110010101100100
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
This is pretty creepy when you think about it.
Flickr/ thevlue
Flickr/ thevlue
This is pretty creepy when you think about it.
I am a rational biped.
one line is definitely longer
Original image: Olly Moss
We are blind to our minds.
1 2System System
1System
2System
Fast Intuitive
Emotional1SystemSystem
1
2SystemSlow
Deliberative Effortful
Data doesn’t change minds.
Ease
Clear Display
Related Experience
Primed Idea
Good Mood
Feels True
Feels Familiar
Feels Good
Feels Effortless
Feeling confident?
You might have a case of Dunning-Kruger
“The incompetent are often blessed with an inappropriate confidence, buoyed by something that feels to them like knowledge.”
—David Dunning
Our access to data has evolved,
our brains haven’t.
Data doesn’t have meaning.
Obviously, what I can
count should
count more.
We have a lot of data. We make a lot of decisions.
We like stories. We aren’t good at math.
The hard decisions aren’t math problems, anyway,
QuantitativeQualitative
What is going on? How are people behaving?
Why? What does it mean?
What is the problem? Any ideas?
Where? When?
How many? How much? How often?
Which yields more?
Usability Studies Field Studies
Interviews
Clickstream Analysis A/B Testing
Surveys*
Are you a statistician?
Flickr/ Billy Wilson
Tight control.
42!Hi!37banana672
No fishing trips.
Know your POV first.
The limit of split testing
Your glorious potential.
What is.
What ought to
be
DecisionsDecisions
What should we do?
Surveys are the most dangerous research tool.
Strongly disagree Disagree Neither agree nor disagree Agree Strongly agree
It is too easy to run a survey.
Easy feels true.
Bad surveys don’t smell bad.
Direct interaction with users is prohibited by my organization, but I have
been allowed to conduct a simple survey by email to identify usability issues.
[a dramatization]
The Nourishing Home / Kelly Smith
My boss is a convert to Foresee. She was apparently very skeptical
of it at first, but she's a very analytical person and was
converted by its promise of being able to quantify unquantifiable data
—like "satisfaction".
Customer satisfaction is a lie.
“Changes in customers’ satisfaction levels explain less than 1% of the variation in changes in their share of category spending.”
—MIT Sloan Management Review
“Customer-service scores have no relevance to stock market returns… the most-hated companies perform better than their beloved peers.”
—Bloomberg Businessweek
“Over a one year period 6% of ‘extremely satisfied customers’ closed their accounts whilst 5.8% of ‘unsatisfied customers’ closed their accounts.”
—HBR, 1995
Will the people I’m surveying be willing and able to provide a truthful answer to my question?
Never ask people
what they like.
Never ask people
to remember.
Never ask people
to predict.
How likely are you to attend the opera?
Very!
How concerned are you about looking smart?
Very!
How likely is it you are telling the truth?
Not Very!
Lies Poppycock Irrelevancies
ONLY WHEN YOU KNOW THE
QUESTION, WILL YOU KNOW WHAT THE ANSWER MEANS.
“At its core, all business is about making bets on human behavior.”
—Ben Wiseman, WSJ
Define success
first.
The most measurable data is not the most
valuable.
Get actionable insights.
Get useful
information.
Make better choices.
We are imperfect humans
designing things for other imperfect humans.
Know your reason.
What and Why
Flickr/johnwiechecki