Statistical inference: Statistical Power, ANOVA, and Post Hoc tests
Post Hoc Tests
description
Transcript of Post Hoc Tests
Post Hoc Tests
What is a Post Hoc Test?
Review:– Adjusting Alpha Level– Multiple A Priori Comparisons
What makes a test Post Hoc?– Many tests could be Post Hoc… But, there are set
Post Hoc tests
Studentized Range Statistic q
2tq
Studentized Range Statistic q
nMS
yyq
error
SLr
Independent Groups
Smallest mean
Largest meanLy
Sy
Example
8.2 8.2 11.8
1y 2y 3y
5n
12errordf
12errorMS
1# stepsr
06.275.1
6.3
54.15
2.88.113
q
)05.,12,3( dfrqcritical value
= 3.77
Fail to RejectNote: arrange means in ascending order!
Studentized Range Statistic q
q’s can tell us where differences are (more specific than F)
Solving q’s is just like solving t’s
But we solve a lot of q’s… can we speed things up?
Solving for the Smallest Significant Difference
n
MS
yyq
error
SLr
n
MSqyy errorrSL
5
4.1577.3 61.6
Example
8.2 8.2 11.8
1y 2y 3y
)05.,12,3( dfrqcritical value
= 3.77
5n
12errorMS
Solving for the Smallest Significant Difference
Solving for the smallest significant difference will help us make quicker comparisons
But we still need a way to organize things nicely…
Newman-Kewls
T1 T2 T3 r
T1 0 3.6 3 6.61
T2 3.6 2 5.41
T3
r
smallest difference required was 6.61
If 2 steps
5
4.1508.32 SL yy 41.5
r smallest significant difference
61.63
41.52
8.2 8.2 11.8
1y 2y 3y
Example
A Better Newman-Kewls Example
Example
2 3 3 9 10
1y 2y 3y 4y 5y
9MSerror 9n
40df
)1(86.22 )1(44.33
)1(79.34 )1(04.45
A Better Newman-Kewls Example
T1 T2 T3 T4 T5 r
T1 1 1 7 8 5 4.04
T2 0 6 7 4 3.79
T3 6 7 3 3.44
T4 1 2 2.86
T5
r T1 T2 T3 T4 T5
T1 * *
T2 * *
T3 * *
T4
T5
Read Right to Left UNTIL
1. The row is completed2. A nonsignificant difference is found
3. Reaching a column which was nonsignificant on the previous row
Newman-Kewls Summarized
Newman-Kewls tables help organize your q’s
When doing a set of post hoc comparisons it’s best to use a Newman-Kewls table
Unequal N’s
n
MS
yyq
error
SLr
n
MSqyy errorrSL
Tukey-Kramer
n
MSerror
2S
error
L
error
nMS
nMS
Replace
with
ny larger L
ny smaller S
Unequal N’s
Behrens-Fisher
),(05.0 fdrSLr qyy 2
22
S
S
L
L
nS
nS
11
2222
222
S
S
S
L
L
L
S
S
L
L
n
nS
n
nS
nS
nS
fd*
Each particular pairing of means must be examined with a different critical value and their own
q2S
Thus, the smallest significant difference will vary even for a given
*
r
A Problem with q So Far…
Why are we doing q’s anyway? Why not do t’s instead?
But is q really controlling our alpha level?
NO!
Using q’s will give use more Type I Errors
Trying to fix q
Tukey's HSD
N-K except
(largest) always rHSD qq
If there are 4 means, all differences are treated as 4 steps.
2rk
WSD
qqq
means of # kr = # of steps between the two
means to be compared.
Tukey's WSD
What Happens to Alpha Level? Power?
Tukey’s HSD and WSD
T1 T2 T3 T4 T5 r
T1 1 1 7 8 5 4.04
T2 0 6 7 4 3.79
T3 6 7 3 3.44
T4 1 2 2.86
T5
r
Use Tukey’s WSD, not normal method for q
Back to Post Hoc in General
What is a post hoc test again?– What are the real issues with Post Hoc tests?
Alpha and Power…
q is just one type of post hoc (one way to balance alpha and power), what are others?
Dunnett’s Control vs. Treatment
run standard t and use )(MS Table edt or, solve for critical difference (CV)
n
MStCV edyy
jTc
2 )(
11 , 30 , 4 , 8 , 1021
nMSyyy eTTc
Go to Table for dt ),( edfk
32.2dt
11
)30(232.2CV
42.5)34.2(32.2
21 Tc yy
62 Tc yy 05.0p
ns
*
Example
Pros…? Cons…?
Sheffé’s Test
Linear contrast
MS(contrast)
MS(error)
F =
To evaluate
1) consult F table and find critical value F.05 (k-1, dferror) (CV)
2) multiply CV by (k-1). (new CV)
It sets the family-wise Type-I Error rate ( in our case) for ALL possible linear contrasts, not merely the pair-wise comparisons.
05.
Don’t use when only doing pair-wise, because it will be overly conservative.
Post Hoc Summary
When to use what…– q in most situations… but use Tukey’s WSD for critical
value Put things in a Newman-Kewls table when N’s are unequal, use Tukey’s correction
– Dunnett’s when you have one control and multiple treatments
– Sheffé’s ONLY when you are doing complex comparisons (i.e., contrasts)
Post Hoc Summary
Be aware of the alpha level and power issues…– Why can’t we have a perfect test (i.e., low alpha
level and high power)?
– How does Tukey’s WSD and HSD relate to this?– How does Dunnett’s relate to this?– How does Sheffé’s relate to this?