Testing Differences Among Several Sample Means Multiple t Tests vs. Analysis of Variance.
-
date post
18-Dec-2015 -
Category
Documents
-
view
213 -
download
1
Transcript of Testing Differences Among Several Sample Means Multiple t Tests vs. Analysis of Variance.
Testing Differences Among Several Sample Means
Multiple t Tests vs. Analysis of Variance
Several Sample Means
• What might we do if we had more than two samples?
X11
X12
.
.
.X1n
X21
X22
.
.
.X2n
Sample 1 Sample 2
€
X 1
€
X 2means:
X31
X32
.
.
.X3n
Sample 3
€
X 3
Several Sample Means
• Specifically how many t Tests could you do?
X11
X12
.
.
.X1n
X21
X22
.
.
.X2n
Sample 1 Sample 2
€
X 1
€
X 2means:
X31
X32
.
.
.X3n
Sample 3
€
X 3
Several Sample Means
• Specifically how many t Tests could you do?
X11
X12
.
.
.X1n
X21
X22
.
.
.X2n
Sample 1 Sample 2
€
X 1
€
X 2means:
X31
X32
.
.
.X3n
Sample 3
€
X 3
Several Sample Means
• Specifically how many t Tests could you do?
X11
X12
.
.
.X1n
X21
X22
.
.
.X2n
Sample 1 Sample 2
€
X 1
€
X 2means:
X31
X32
.
.
.X3n
Sample 3
€
X 3
Several Sample Means
• Specifically how many t Tests could you do?
X11
X12
.
.
.X1n
X21
X22
.
.
.X2n
Sample 1 Sample 2
€
X 1
€
X 2means:
X31
X32
.
.
.X3n
Sample 3
€
X 3
Multiple t Tests and Family-wise Error Rate
• If you do all possible pair-wise comparisons (C), what happens to the overall probability of making a Type I error?
Multiple t Tests and Family-wise Error Rate
• If you do all possible pair-wise comparisons (C), what happens to the overall probability of making a Type I error?
€
α(C)Family-wise Error Rate =
Multiple t Tests and Family-wise Error Rate
• How could we prevent the family-wise error rate from exceeding .05 ?
Multiple t Tests and Family-wise Error Rate
• How could we prevent the family-wise error rate from exceeding .05 ?
• Set the alpha-level for each pair-wise t test to be a fraction of .05; specifically:
€
α pair =α family
C
Multiple t Tests and Familywise Error Rate
• This isn’t usually done in practice because only a few different alpha-levels appear in the t tables
Multiple t Tests and Familywise Error Rate
• This isn’t usually done in practice because only a few different alpha-levels appear in the t tables
• More importantly, consider that C increases dramatically as more samples are added– for 4 samples: C = 6– for 5 samples: C = 10– for 6 samples: C = 15
• Which leads to a precipitous drop in power
Analysis of Variance
• What is needed is a technique that controls family-wise error rate while looking for one or more differences between several sample means
Analysis of Variance
• What is needed is a technique that controls family-wise error rate while looking for one or more differences between several sample means
• That technique is a one-way Analysis of Variance (ANOVA)
Analysis of Variance• Here are three samples, each are
measurements under different treatment conditions:
X11
X12
.
.
.X1n
X21
X22
.
.
.X2n
Sample 1 Sample 2
€
X 1
€
X 2means:
X31
X32
.
.
.X3n
Sample 3
€
X 3
Each sample has a mean and variance and the 3 means are a sampling distribution of means
Analysis of Variance
• What would the null hypothesis be?
• What would the alternative hypothesis be?
X11
X12
.
.
.X1n
X21
X22
.
.
.X2n
Sample 1 Sample 2
€
X 1
€
X 2means:
X31
X32
.
.
.X3n
Sample 3
€
X 3
Analysis of Variance• What would the null hypothesis be?
– All three samples are taken from the same population so:
€
μ1 = μ2 = μ3 = μ
Analysis of Variance• What would the null hypothesis be?
– All three samples are taken from the same population so:
• What would the alternative hypothesis be?– At least one of the samples is from a different population and hence has a different mean
€
μ1 = μ2 = μ3 = μ
Analysis of Variance• We can estimate the variance of the “null hypothesis”
population by averaging the j variance estimates
Analysis of Variance
• This is called the “Mean Square Error” or “Mean Square Within”
€
MSerror =ˆ σ 1
2 + ˆ σ 22 + ˆ σ 3
2
3€
MSerror =
ˆ σ j2
j=1
k
∑
kin our example:
Analysis of Variance
• MSerror is an estimate of the population variance
€
MSerror =ˆ σ 1
2 + ˆ σ 22 + ˆ σ 3
2
3
Analysis of Variance• MSerror is an estimate of the population variance
• What’s another way we could estimate the population variance (hint: assume the null hypothesis is true)?
€
MSerror =ˆ σ 1
2 + ˆ σ 22 + ˆ σ 3
2
3
Analysis of Variance• Each sample has a mean and variance and the
3 means are a sampling distribution of means
X11
X12
.
.
.X1n
X21
X22
.
.
.X2n
Sample 1 Sample 2
€
X 1
€
X 2means:
X31
X32
.
.
.X3n
Sample 3
€
X 3
Analysis of Variance• Recall that we estimated the variance of a sampling distribution of
means (since we only had one sample) using the equation:
€
ˆ σ X 2 =
ˆ σ 2
n
Analysis of Variance• Now we’ve got more than one sample! So we can turn this
equation around and make an estimate of the population variance called the “Mean Square Effect” or “Mean Square Between”:
€
MSeffect = ˆ σ 2 = n ˆ σ X 2 = n
(X j − X overall )2
j=1
k
∑
k −1€
ˆ σ X 2 =
ˆ σ 2
n
Analysis of Variance
• We now have two different estimates of the population variance: MSerror and MSeffect
• Why might these two estimates disagree?
Analysis of Variance
• MSerror is based on deviation scores within each sample but…
Analysis of Variance
• MSerror is based on deviation scores within each sample but…
• MSeffect is based on deviations between samples
Analysis of Variance
• MSerror is based on deviation scores within each sample but…
• MSeffect is based on deviations between samples
• MSeffect would overestimate the population variance when…
Analysis of Variance
• MSerror is based on deviation scores within each sample but…
• MSeffect is based on deviations between samples
• MSeffect would overestimate the population variance when…there is some effect of the treatment pushing the means of the different samples apart
Analysis of Variance
• We compare MSeffect against MSerror by constructing a statistic called F
Analysis of Variance
• We compare MSeffect against MSerror by constructing a statistic called F
• If the hull hypothesis:
is true then we would expect:
except for random sampling variation €
μ1 = μ2 = μ3 = μ
€
X 1 = X 2 = X 3 = μ
Analysis of Variance
• F is the ratio of MSeffect to MSerror
€
Fk−1,k(n−1) =MSeffect
MSerror
Analysis of Variance
• F is the ratio of MSeffect to MSerror
• If the null hypothesis is true then F should equal 1.0
€
Fk−1,k(n−1) =MSeffect
MSerror
Analysis of Variance
• Of course there is a sampling distribution of F - if you repeated your experiment many times you would get a distribution of Fs
Analysis of Variance
• Of course there is a sampling distribution of F - if you repeated your experiment many times you would get a distribution of Fs
• The shape of that distribution depends on two different degrees of freedom:– MSeffect has k-1 degrees of freedom
– MSerror has k(n-1) degrees of freedom
Analysis of Variance
• We can look up a critical F from an F table for any given number of degrees of freedom
Analysis of Variance
• We can look up a critical F from an F table for any given number of degrees of freedom
• If the F statistic we’ve obtained in our experiment exceeds Fcrit then we know that fewer than 5% of such experiments would be likely to obtain this F statistic if the null hypothesis was true
Analysis of Variance
• We can look up a critical F from an F table for any given number of degrees of freedom
• If the F statistic we’ve obtained in our experiment exceeds Fcrit then we know that fewer than 5% of such experiments would be likely to obtain this F statistic if the null hypothesis was true
• So we can reject the null and conclude that at least one pair of means is different