Post on 16-Jan-2015
description
Web Directions User Experience ‘08
Analysing quantitative data
with Steve BatyUX Strategist
Web Directions User Experience ’08 - Analysing Quantitative Data
Data is important
Web Directions User Experience ’08 - Analysing Quantitative Data
We expend a lot of effort to gather it
Web Directions User Experience ’08 - Analysing Quantitative Data
We don’t always use it well
Web Directions User Experience ’08 - Analysing Quantitative Data
We’ll be looking at:
Web Directions User Experience ’08 - Analysing Quantitative Data
We’ll be looking at:* time-to-completion
Web Directions User Experience ’08 - Analysing Quantitative Data
We’ll be looking at:
* task completion rates* time-to-completion
Web Directions User Experience ’08 - Analysing Quantitative Data
We’ll be looking at:
* a/b testing* task completion rates* time-to-completion
Web Directions User Experience ’08 - Analysing Quantitative Data
We’ll be looking at:
* page-view data* a/b testing
* task completion rates* time-to-completion
Web Directions User Experience ’08 - Analysing Quantitative Data
time-to-completion
Web Directions User Experience ’08 - Analysing Quantitative Data
Web Directions User Experience ’08 - Analysing Quantitative Data
1 min 24 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
1 min 23.8 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
1 min 23.77 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
1 min 23.768 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
83.768 secs
Web Directions User Experience ’08 - Analysing Quantitative Data
User 1 User 2 User...Task 1Task 2Task 3Task 4Task 5Task 6Task 7
83.5 97.3131.1 165.554.5 45.597.8 88.2
118.0 143.3243.9 309.022.9 23.9
Our data might
look like this...
Web Directions User Experience ’08 - Analysing Quantitative Data
We can calculate...
Web Directions User Experience ’08 - Analysing Quantitative Data
We can calculate...mean - AVERAGE()variance - VAR()standard dev’n - STDEV()
Web Directions User Experience ’08 - Analysing Quantitative Data
Low-variability
Medium-variability
High-variability
90.43 s
Web Directions User Experience ’08 - Analysing Quantitative Data
Compare 2 sets of data- between iterations- between audience segments
Web Directions User Experience ’08 - Analysing Quantitative Data
Low sample sizes restrict options
Web Directions User Experience ’08 - Analysing Quantitative Data
non-parametric version == no assumed dist’n
Web Directions User Experience ’08 - Analysing Quantitative Data
Rank-sum test
Web Directions User Experience ’08 - Analysing Quantitative Data
Time for a practical demonstration
Web Directions User Experience ’08 - Analysing Quantitative Data
Web Directions User Experience ’08 - Analysing Quantitative Data
Web Directions User Experience ’08 - Analysing Quantitative Data
1 3 3 3 5.5 5.5 7.5 7.5 9 10
Web Directions User Experience ’08 - Analysing Quantitative Data
1 3 3 3 5.5 5.5 7.5 7.5 9 10
2+3+4=9/3
}5+6
=11/2
}7+8
=15/2
}
Web Directions User Experience ’08 - Analysing Quantitative Data
1 3
3
3
5.5
5.5
7.5
7.5
10
9
m = 5S1 = 28
S0 = 27n = 5
Web Directions User Experience ’08 - Analysing Quantitative Data
U0 = nm +n n +1( )2
⎡⎣⎢
⎤⎦⎥− S0
= 5x5 +5 5 +1( )2
⎡⎣⎢
⎤⎦⎥− 27
= 13
Web Directions User Experience ’08 - Analysing Quantitative Data
U0 = nm +n n +1( )2
⎡⎣⎢
⎤⎦⎥− S0
= 5x5 +5 5 +1( )2
⎡⎣⎢
⎤⎦⎥− 27
= 13
90% --> 3895% --> 4199% --> 45
Web Directions User Experience ’08 - Analysing Quantitative Data
task completion rates
Web Directions User Experience ’08 - Analysing Quantitative Data
Only 2 possible values: success or fail
Web Directions User Experience ’08 - Analysing Quantitative Data
Small samples lead to very broad estimates
Web Directions User Experience ’08 - Analysing Quantitative Data
4/6 successes = 66.67%
21% - 99.3% with 62.5% most likely
Web Directions User Experience ’08 - Analysing Quantitative Data
With 30 users47.7% - 81.9% with 64.8% most likely
Web Directions User Experience ’08 - Analysing Quantitative Data
Most likely =
Range = p ± zp 1− p( )
n
p =s +1n + 2
Web Directions User Experience ’08 - Analysing Quantitative Data
p ± zp 1− p( )
n
Web Directions User Experience ’08 - Analysing Quantitative Data
p ± zp 1− p( )
nmost likely
Web Directions User Experience ’08 - Analysing Quantitative Data
p ± zp 1− p( )
n
confidencelevel
Web Directions User Experience ’08 - Analysing Quantitative Data
p ± zp 1− p( )
n
variability
Web Directions User Experience ’08 - Analysing Quantitative Data
A/B Testing
Photo courtesy of www.dorothyphoto.com
Web Directions User Experience ’08 - Analysing Quantitative Data
Compare two different approaches to the
same problem
Web Directions User Experience ’08 - Analysing Quantitative Data
Run both simultaneously;
randomly divert users to option B
Web Directions User Experience ’08 - Analysing Quantitative Data
Compare using a Chi-squared test
Web Directions User Experience ’08 - Analysing Quantitative Data
Example: clicks on an ad banner
Ignore Click Total
A
B
10,119 275 10,394
962 38 1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data
χ 2 =eij − oij( )2eij
∑The test statistic is a measure of distance
between what we expect to see (e), and what we actually observed (o). For each cell, subtract what we expect from what we saw, square it to remove any negative values, and divide it by
the expected value. Add it all together...
Web Directions User Experience ’08 - Analysing Quantitative Data
Calculated expected valuesFor each cell:
row total x column total/grand total
Web Directions User Experience ’08 - Analysing Quantitative Data
Ignore Click Total
A
B
10,108 = 10,394x(11,081/11,394)
286 = 10,394x(313/11,394) 10,394
973 = 1,000x(11,081/11,394)
27 = 1,000x(313/11,394) 1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data
Ignore Click Total
A
B
10,108 - 10,119 = -11 286 - 275 = 11 10,394
973 - 962 = 11 27 - 38 = -11 1,000
Total 11,081 313 11,394
Web Directions User Experience ’08 - Analysing Quantitative Data
χ 2 =eij − oij( )2eij
∑
=112
10,108+112
286+112
973+112
27= 0.012 + 0.423+ 0.124 + 4.48= 5.04
Web Directions User Experience ’08 - Analysing Quantitative Data
χα =0.0252 = 5.02 < χ 2
χα =0.012 = 6.63 > χ 2
Web Directions User Experience ’08 - Analysing Quantitative Data
page viewspre- & post comparison
Web Directions User Experience ’08 - Analysing Quantitative Data
Can be cyclical
Web Directions User Experience ’08 - Analysing Quantitative Data
Can be cyclical
Web Directions User Experience ’08 - Analysing Quantitative Data
Can be trending
Web Directions User Experience ’08 - Analysing Quantitative Data
Typically compare the average
Web Directions User Experience ’08 - Analysing Quantitative Data
But ignores fluctuation
Web Directions User Experience ’08 - Analysing Quantitative Data
But ignores fluctuation
?
Web Directions User Experience ’08 - Analysing Quantitative Data
z =x1 − x2( )s12
n1+ s2
2
n2Test 1 : x1, s1
2 ,n1Test 2 : x2 , s2
2 ,n2
Web Directions User Experience ’08 - Analysing Quantitative Data
z =x1 − x2( )s12
n1+ s2
2
n2Test 1 : x1, s1
2 ,n1Test 2 : x2 , s2
2 ,n2
In order: mean, variance &
number of data points in each
test.
Web Directions User Experience ’08 - Analysing Quantitative Data
z =x1 − x2( )s12
n1+ s2
2
n2Test 1 : x1, s1
2 ,n1Test 2 : x2 , s2
2 ,n2
In order: mean, variance &
number of data points in each
test.
Mean difference
Web Directions User Experience ’08 - Analysing Quantitative Data
z =x1 − x2( )s12
n1+ s2
2
n2Test 1 : x1, s1
2 ,n1Test 2 : x2 , s2
2 ,n2
In order: mean, variance &
number of data points in each
test.
Mean difference
Combined standard error
Web Directions User Experience ’08 - Analysing Quantitative Data
z =x1 − x2( )s12
n1+ s2
2
n2Test 1 : x1, s1
2 ,n1Test 2 : x2 , s2
2 ,n2If z < -1.96 or > 1.96 a significance difference exists
In order: mean, variance &
number of data points in each
test.
Mean difference
Combined standard error
Web Directions User Experience ’08 - Analysing Quantitative Data
xs2ni
Pre Post
1,288 1,331
1,369 756.25
60 30
Web Directions User Experience ’08 - Analysing Quantitative Data
z =x1 − x2( )s12
n1+ s2
2
n2
=1288 −1331( )136960
+ 756.2530
=436.93
= 6.205
Web Directions User Experience ’08 - Analysing Quantitative Data
1,288 1,331
Web Directions User Experience ’08 - Analysing Quantitative Data
Read more...Statistics without tears by Derek Rowntree
Flaws & Fallacies in statistical thinking by Stephen K Campbell
http://uxstats.blogspot.com
Web Directions User Experience ’08 - Analysing Quantitative Data
Thank you