Comparing Cell Means
description
Transcript of Comparing Cell Means
Comparing Cell Means
Planned Comparisons & Post Hoc Tests
Questions What is the main difference between planned
comparisons and post hoc tests? Generate numbers (like 0 1, -1 or 1 –1/2, -1/2) to
create a contrast appropriate for a given problem. How many independent comparisons can be made
in a given design? What is the difference between a per comparison
and a familywise error rate? How does Bonferroni deal with familywise error
rate problems? What is the studentized range statistic? How is it
used?
Questions (2) What is the difference between the Tukey HSD
and the Newman-Keuls? What are the considerations when choosing a post
hoc test (what do you need to trade-off)? Describe (make up) a concrete example where you
would use planned comparisons instead of an overall F test. Explain why the planned comparison is the proper analysis.
Describe (make up) a concrete example where you would use a post hoc test. Explain why the post hoc test is needed (not the specific choice of post hoc test, but rather why post hoc test at all).
Planned vs. Post Hoc
• Planned Comparisons or Contrasts– Use instead of overall F test. Planned
before the study.
• Post Hoc or Incidental tests. – Use after significant overall F test to
investigate specific means. No specific plan before study.
Control Comp Tutor
Comp Tutor+
Lab
Comp Tutor + lab + quiz
Planned Comparisons (1)
Population Comparison:
J
jjjJJ cccc
12211 ...
Weights are real numbers not all zero. Sum of weights must equal zero.
J
jjc
1
0
Sample Comparison:
J
jjjJJ ycycycycest
12211 ...ˆ.
J
jjc
1
0
Planned Comparison (2)
A1 A2 A3 A422 26 28 21
15 27 31 21
17 24 27 26
18 23 26 20
18 25 28 22jy
Source SS df MS
F
Cells (A1-A4)
219 3 73 12.17
Error 72 12 6
Total 291 15
(Data)
(Summary Table)
Comparison A1 A2 A3 A4
1 1/2 1/2 -1/2 -1/2
2 1 -1 0 0
3 0 0 1 -1
(3 possible comparisons)
5.3)22*5(.)28*5(.
)25*5(.)18*5(.ˆ1
72518ˆ 2
6)22()28(
)25*0()18*0(ˆ3
Sampling Variance of Planned ComparisonsThe sample comparison is an unbiased estimate of the population comparison.
The variance of the sampling distribution of the comparison:
)ˆ(E
j j
jej
jj n
cycVar
222 )var()ˆ(
Sampling variance will be large when within cells variance is large, the weights are large, and the number of people in each cell is small. Estimated by:
j j
jerror n
cMSVarest
2
)()ˆ(. We substitute for
errorMS2e
Significance TestA1 A2 A3 A422 26 28 21
15 27 31 21
17 24 27 26
18 23 26 20
18 25 28 22
Source SS df MS
F
Cells (A1-A4)
219 3 73 12.17
Error 72 12 6
Total 291 15
j j
jerror n
cMSVarest
2
)()ˆ(.
5.3)22*5(.)28*5(.
)25*5(.)18*5(.ˆ1
2/34
)5.()5.(5.5.6)ˆ(.
2222
Varest
2247.12/3)ˆ(. SEest
86.22247.1
5.3
)ˆvar(.
ˆ
estt
df=N-J; 16-4=12=dfe.
t(12) =-2.86, p < .05
18.2)12,05.,( dfcritt
Significance TestA1 A2 A3 A422 26 28 21
15 27 31 21
17 24 27 26
18 23 26 20
18 25 28 22
Source SS df MS
F
Cells (A1-A4)
219 3 73 12.17
Error 72 12 6
Total 291 15
j j
jerror n
cMSVarest
2
)()ˆ(.
)ˆ(. Varest
)ˆ(. SEest
t
df=N-J=
72518.ˆ 212 AvsA
critt
Review
What is the main difference between planned comparisons and post hoc tests?
Suppose I do a blind orange juice taste test and discover that my means are:
Tropicana Florida Fresh
Pulpmaster
7.3 5.5 6.4
If my hypothesis is that Tropicana is better than all others, what are my contrast weights?
Independence of Planned ComparisonsYou can make several planned comparisons on the same data.
Some of these comparisons are independent; some are dependent. We want them independent. Two comparisons from a normal population with equal sample sizes in each cell are independent if the sum of the products of weights is zero.
j
jjcc 021 With unequal sample sizes, it’s:
j j
jj
n
cc021
Independence (2) j
jjcc 021
Comparison A1 A2 A3 A4
1 -1/3 1 -1/3 -1/3
2 -1/2 0 -1/2 1
3 1/2 1/2 -1/2 -1/2
0)1*3/1()2/1*3/1(
)0*1()2/1*3/1(21
j
jjcc
3/26/4)2/1*3/1()2/1*3/1(
)2/1*1()2/1*3/1(31
j
jjcc
One and two are orthogonal; one and three are not.
There are J-1 orthogonal comparisons. Use only what you need.
Choosing Comparisons
Usually done on basis of theory. But there are methods to generate all possible orthogonal comparisons.
Group
1 2 3 4 5
Comparison 1 4 -1 -1 -1 -1
2 0 3 -1 -1 -1
3 0 0 2 -1 -1
4 0 0 0 1 -1
Error Rates
• With 1 test, we set alpha = Type I error rate.• With multiple tests, original (nominal) alpha
is called the per comparison error rate ( ).• With comparisons, we have a family of tests
on the same data. Want to know the probability of at least 1 Type I error in the family of tests. Such a probability is called familywise error rate ( ).
• For independent tests,• E.g., 10 tests:
PC
FWK
PCFW )1(1
40.6.195.1)05.1(1 1010 FW
Bonferroni Tests
• Familywise error depends on the number of tests (K) and the nominal alpha, .
• Bonferroni’s solution is to set:
• Suppose we want FW error to be .05 and we will have 4 comparisons. Then
KFW
BPC
* PC
Where is an aspiration level.*FW
We use the adjusted alpha (.0125) for each of the 4 tests.
0125.4
05.BPC
Bonferroni Test (2)
• Use the adjusted alpha (e.g., .0125) for each comparison.
• Look at the p value on the printout (use .0125 instead of .05).
• Use a statistical function (e.g., Excel, SAS) if you want to find the critical value.
• E.g., Excel function TINV says with p=.0125 and df=12, t is 2.93.
Review
How many independent comparisons can be made in a given design?
What is the difference between a per comparison and a familywise error rate?
How does Bonferroni deal with familywise error rate problems?
Post Hoc Tests
• Given a significant F, where are the mean differences?
• Often do not have planned comparisons.
• Usually compare pairs of means.
• There are many methods of post hoc (after the fact) tests.
Scheffé
• Can use for any contrast. Follows same calculations, but uses different critical values.
• Instead of comparing the test statistic to a critical value of t, use:
)ˆvar(.
ˆ
estt
FJ )1(S
Where the F comes from the overall F test (J-1 and N-J df).
Scheffé (2) Source SS df MS
F
Cells (A1-A4)
219 3 73 12.17
Error 72 12 6
Total 291 155.3)22*5(.)28*5(.
)25*5(.)18*5(.ˆ1
2/34
)5.()5.(5.5.6)ˆ(.
2222
Varest
86.22247.1
5.3
)ˆvar(.
ˆ
estt
24.349.3)14()1(S FJ
49.3)12,3,05.( F
(Data from earlier problem.)
The comparison is not significant because |-2.86|<3.24.
Paired comparisons
Newman Keuls and Tukey HSD are two (of many) choices. Both depend on q, the studentized range statistic. Suppose we have J independent sample means and we find the largest and the smallest.
nMS
yyq
error /minmax
MSerror comes from the ANOVA we did to get the J means. The n refers to sample size per cell. If two cells are unequal, use 2n1n2/(n1+n2).
The sampling distribution of q depends on k, the number of means covered by the range (max-min), and on v, the degrees of freedom for MSerror.
Tukey HSD
HSD = honestly significant difference. For HSD, use k = J, the number of groups in the study. Choose alpha, and find the df for error. Look up the value qα. Then find the value:
n
MSqHSD error
Compare HSD to the absolute value of the difference between all pairs of means. Any difference larger than HSD is significant.
HSD 2 Grp -> 1 2 3 4 5
M -> 63 82 80 77 70
Source SS df MS F p
Grps 2942.4 4 725.6 4.13 <.05
Error 9801.0 55 178.2
K = 5 groups; n=12 per group, v has 55 df. Tabled value of q with alpha =.05 is 3.98.
34.1512
2.17898.3
n
MSqHSD error
Group 1 5 4 3 2
1 63 0 7 14 17* 19*
5 70 0 7 10 12
4 77 0 3 5
3 80 0 2
2 82 0
Newman-KeulsGroup 1 2 3 4 5
1 63 0 7 14* 17* 19*
2 70 0 7 10 12
3 77 0 3 5
4 80 0 2
5 82 0Layer 1
Layer 2Layer 3
Layer 4
Same as HSD except the value of q changes with layers. For layer k-1 (here 4), use HSD. For each layer down, subtract 1 from the value of k for the tabled value of q.
34.1512
2.17898.34
n
MSqNK error
40.14
12
2.17874.33 NK
09.1312
2.1784.32 NK 90.10
12
2.17883.21 NK
Layer refers to how many means apart.
Comparing Post Hoc Tests
The Newman-Keuls found 3 significant differences in our example. The HSD found 2 differences. If we had used the Bonferroni approach,we would have found an interval of 15.91 required for significance (and therefore the same two significant as HSD). Thus, power descends from the Newman-Keuls to the HSD to the Bonferroni. The type I error rates go just the opposite, the lowest to Bonferroni, then HSD and finally Newman-Keuls. Do you want to be liberal or conservative in your choice of tests? Type I error vs Power.
Review What is the studentized range statistic? How is it used? What is the difference between the Tukey HSD and the
Newman-Keuls? What are the considerations when choosing a post hoc
test (what do you need to trade-off)? Describe (make up) a concrete example where you
would use planned comparisons instead of an overall F test. Explain why the planned comparison is the proper analysis.
Describe (make up) a concrete example where you would use a post hoc test. Explain why the post hoc test is needed (not the specific choice of post hoc test, but rather why post hoc test at all).