Aims and Objectives - Discovering Statistics
Transcript of Aims and Objectives - Discovering Statistics
11
Meta-AnalysisMetaMeta--AnalysisAnalysis
Dr. Andy FieldDr. Andy FieldDr. Andy Field
7-Mar-03 Andy Field Slide 2
Aims and ObjectivesAims and ObjectivesAims and Objectives
When and Why?
Theory• Fixed Effects Models
• Random Effects Models
Can meta-analysis be trusted?• Practical problems
• Choice of methods
When and Why?When and Why?
TheoryTheory
•• Fixed Effects ModelsFixed Effects Models
•• Random Effects ModelsRandom Effects Models
Can metaCan meta--analysis be trusted?analysis be trusted?
•• Practical problemsPractical problems
•• Choice of methodsChoice of methods
7-Mar-03 Andy Field Slide 3
When and Why?When and Why?When and Why?To assimilate information from independent studies
Discursive Literature Reviews• Selective inclusion of studies
• Subjective weighting of studies
• Biased interpretation of evidence
Meta-Analysis• Objective assimilation of studies?
To assimilate information from To assimilate information from independent studiesindependent studies
Discursive Literature ReviewsDiscursive Literature Reviews•• Selective inclusion of studiesSelective inclusion of studies
•• Subjective weighting of studiesSubjective weighting of studies
•• Biased interpretation of evidenceBiased interpretation of evidence
MetaMeta--AnalysisAnalysis•• Objective assimilation of studies?Objective assimilation of studies?
7-Mar-03 Andy Field Slide 4
Use of MetaUse of Meta--Analysis (1981Analysis (1981--2001)2001)
Year
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 2000 2001
Num
ber o
f Pub
licat
ions
0
100
200
300
400
500
600
700
800
900
1000
1100
1200
1300
In Social Science JournalsIn Science Journals
7-Mar-03 Andy Field Slide 5
What do scientists do?What do scientists do?What do scientists do?Interested in knowing a ‘true’ effect size in our population of interest.Use samples to estimate this true effectExpress the effect:• Glass’s ∆• Cohen’s d• Hedge’s g• Pearson’s r• Odds ratios/risk rates
Discover that others have also tried to find this effectAssimilate these different studies.
Interested in knowing a ‘true’ effect size in our Interested in knowing a ‘true’ effect size in our population of interest.population of interest.Use samples to estimate this true effectUse samples to estimate this true effectExpress the effect:Express the effect:•• Glass’s Glass’s ∆∆•• Cohen’s Cohen’s dd•• Hedge’s Hedge’s gg•• Pearson’s Pearson’s rr•• Odds ratios/risk ratesOdds ratios/risk rates
Discover that others have also tried to find this effectDiscover that others have also tried to find this effectAssimilate these different studies.Assimilate these different studies.
7-Mar-03 Andy Field Slide 6
Meta-Analysis: BasicsMetaMeta--Analysis: BasicsAnalysis: BasicsCalculate effect sizes for each study• Convert to a common metric
Weight each effect size by the sampling precision• Sample size (or some function of it)
Calculate the mean effect size across studies
Express this mean as a Z-score• Calculate significance/confidence intervals
Moderator Analysis?
Calculate effect sizes for each studyCalculate effect sizes for each study•• Convert to a common metricConvert to a common metric
Weight each effect size by the sampling Weight each effect size by the sampling precisionprecision•• Sample size (or some function of it)Sample size (or some function of it)
Calculate the mean effect size across studiesCalculate the mean effect size across studies
Express this mean as a ZExpress this mean as a Z--scorescore•• Calculate significance/confidence intervalsCalculate significance/confidence intervals
Moderator Analysis?Moderator Analysis?
22
7-Mar-03 Andy Field Slide 7
r = 0.29 r = 0.30 r = 0.42
Fixed Effects ModelsFixed Effects Modelsρ = 0.30
r = 0.22
7-Mar-03 Andy Field Slide 8
ρ = 0.25 ρ = 0.30
r = 0.29 r = 0.30r = 0.42
Random Effects Random Effects ModelsModels Superpopulation
ρ = 0.30
r = 0.22
ρ = 0.27 ρ = 0.36
7-Mar-03 Andy Field Slide 9
Hedges et al.’s Fixed MethodHedges Hedges et alet al.’s Fixed Method.’s Fixed Method
( )i
i
i rr
erz −+= 11
21 log
∑
∑=
=
=k
ii
k
iiri
w
zw
rz1
1
iviw 1=
31−=
iniv
3−=∴ ii nw
( )∑
∑=
=
=
−
−
k
ii
k
iiri
n
zn
rz1
1
3
)3(
Mean Effect Size:Mean Effect Size:Mean Effect Size:( )( ) 1
12
2
+−=
iz
iz
ee
ir
7-Mar-03 Andy Field Slide 10
Significance of Mean Effect Size:Significance of Mean Effect Size:Significance of Mean Effect Size:
( )∑
==
k
iiw
rzSE1
1 ( )( )∑
==
−k
iin
rzSE1
3
1
( )r
rzSE
zZ =
7-Mar-03 Andy Field Slide 11
Homogeneity of Effect Sizes:Homogeneity of Effect Sizes:Homogeneity of Effect Sizes:
( )( )∑=
−−=k
irri zznQ
i1
23
7-Mar-03 Andy Field Slide 12
Hedges et al.’s Random MethodHedges Hedges et alet al.’s Random Method.’s Random Method
∑
∑=
=
=k
ii
k
iiri
w
zw
rz1
*
1
*2
1*
τ+=
iviw
31−=
iniv
12*
31
−
+
−=∴ τ
ii nw
∑
∑=
=
−=
−
+
−
+
−k
i i
k
iir
i
n
zn
rz1
12
1
12
31
31
τ
τ
Mean Effect Size:Mean Effect Size:Mean Effect Size:
33
7-Mar-03 Andy Field Slide 13
Estimating Between-Study VarianceEstimating BetweenEstimating Between--Study VarianceStudy Variance
( )ckQ 12 −−=τ
( )
∑
∑−=
=
=∑=
k
ii
k
ii
w
wk
iiwc
1
1
2
1
( )( )
( )∑
∑−−=
=
=
−
−
=∑ k
ii
k
ii
n
nk
iinc
1
1
2
3
3
1
3
7-Mar-03 Andy Field Slide 14
Hunter-Schmidt MethodHunterHunter--Schmidt MethodSchmidt Method
Mean Effect Size:Mean Effect Size:Mean Effect Size:
∑
∑=
=
=k
ii
k
iii
n
rn
r1
1
7-Mar-03 Andy Field Slide 15
Significance of Mean Effect Size:Significance of Mean Effect Size:Significance of Mean Effect Size:
( )
∑
∑=
=
=
−
k
ii
k
iii
n
rrn
rSD1
1
2
kSD
rrSE =
rSErZ =
7-Mar-03 Andy Field Slide 16
Homogeneity of Effect Sizes:Homogeneity of Effect Sizes:Homogeneity of Effect Sizes:
( )( )( )∑
=−
−−=k
ir
rrn ii
1 1
1222
2
χ
7-Mar-03 Andy Field Slide 17
Can meta-analysis be trusted?Can metaCan meta--analysis be trusted?analysis be trusted?
Publication bias: the ‘file drawer’ problem• Significant findings more often published
• 97% of published psychology articles report significant results (Sterling, 1959)
• Meta-analysis will overestimate the mean effect size
Artefacts• Is all research equally good?
Publication bias: the ‘file drawer’ problemPublication bias: the ‘file drawer’ problem
•• Significant findings more often publishedSignificant findings more often published
•• 97% of published psychology articles report 97% of published psychology articles report significant results (Sterling, 1959)significant results (Sterling, 1959)
•• MetaMeta--analysis will overestimate the mean effect analysis will overestimate the mean effect sizesize
ArtefactsArtefacts
•• Is all research equally good?Is all research equally good?
7-Mar-03 Andy Field Slide 18
Which method: Fixed or Random?
Which method: Fixed or Which method: Fixed or Random?Random?
Real data are likely to have heterogeneous effect sizes:
However, Fixed methods are more commonly applied:• Hunter & Schmidt (2001)
– No random effects meta-analyses in Psychological Bulletin
• Field (2003): Bias when fixed effects methods are used on heterogeneous effect sizes.
Real data are likely to have heterogeneous Real data are likely to have heterogeneous effect sizes:effect sizes:
However, Fixed methods are more commonly However, Fixed methods are more commonly applied:applied:
•• Hunter & Schmidt (2001)Hunter & Schmidt (2001)
–– No random effects metaNo random effects meta--analyses in Psychological Bulletinanalyses in Psychological Bulletin
•• Field (2003): Bias when fixed effects methods are Field (2003): Bias when fixed effects methods are used on heterogeneous effect sizes.used on heterogeneous effect sizes.
44
7-Mar-03 Andy Field Slide 19
Field (2003)Field (2003)Field (2003)
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.70
0.75
0.80
202020202020404040404040808080808080
16016016016016016051015202530
51015202530
51015202530
51015202530
Typ
e I
Err
or
Rate
Sample Size
Studies in Meta-Analysis
Fixed-Effects Models Applied toHeterogeneous Effect Sizes
Hedges & Colleagues
Rosenthal-Rubin
7-Mar-03 Andy Field Slide 20
Which Method: Hedges or Hunter-Schmidt?
Which Method: Hedges or Which Method: Hedges or HunterHunter--Schmidt? Schmidt?
Transforming r-to-z
• Corrects for bias in sampling distribution of r
• Transforming r is a good thing (Silver & Dunlap (1987)
• Transforming r makes no difference (Strube, 1988; Hunter et al., 1996)
Transforming Transforming rr--toto--zz
•• Corrects for bias in sampling distribution Corrects for bias in sampling distribution of of rr
•• Transforming Transforming rr is a good thing (Silver & is a good thing (Silver & Dunlap (1987)Dunlap (1987)
•• Transforming Transforming rr makes no difference makes no difference ((StrubeStrube, 1988; Hunter , 1988; Hunter et alet al., 1996)., 1996)
7-Mar-03 Andy Field Slide 21
Estimates of Standard ErrorEstimates of Standard ErrorEstimates of Standard Error
Hedges & Vevea (1998)• Hunter-Schmidt use suboptimal weights
• Hence, when between study variance exists H-S will underestimate SE and overestimate Z (Type I errors).
Field (2001):• Hedges & Vevea’s method uses truncated
estimates of between-study variance.
• Hence biased when the number of studies in the meta-analysis are small.
Hedges & Hedges & VeveaVevea (1998)(1998)•• HunterHunter--Schmidt use suboptimal weightsSchmidt use suboptimal weights
•• Hence, when between study variance exists HHence, when between study variance exists H--S S will underestimate SE and overestimate Z (Type I will underestimate SE and overestimate Z (Type I errors).errors).
Field (2001):Field (2001):•• Hedges & Hedges & Vevea’sVevea’s method uses truncated method uses truncated
estimates of betweenestimates of between--study variance.study variance.
•• Hence biased when the number of studies in the Hence biased when the number of studies in the metameta--analysis are small.analysis are small.
7-Mar-03 Andy Field Slide 22
Comparing MethodsComparing MethodsComparing MethodsJohnson et al. (1995)• Compared Hedges (fixed) method with H-S by
manipulating a single data set.
• H-S provided conservative estimates of significance
Field (2001):• Johnson et al.’s study is pants:
– Results could be due to the data set used
– They got the H-S equations wrong
– Used Hedges d method and converted to r
– Only looked at fixed-effect data
Johnson et al. (1995)Johnson et al. (1995)
•• Compared Hedges (fixed) method with HCompared Hedges (fixed) method with H--S by S by manipulating a single data set.manipulating a single data set.
•• HH--S provided conservative estimates of significanceS provided conservative estimates of significance
Field (2001):Field (2001):
•• Johnson et al.’s study is pants:Johnson et al.’s study is pants:
–– Results could be due to the data set usedResults could be due to the data set used
–– They got the HThey got the H--S equations wrongS equations wrong
–– Used Hedges Used Hedges dd method and converted to method and converted to rr
–– Only looked at fixedOnly looked at fixed--effect dataeffect data
7-Mar-03 Andy Field Slide 23
Field (2001)Field (2001)Field (2001)Monte Carlo Study• Effect sizes sampled from a population with a fixed
or variable mean effect size.
• For each: 100,000 repetitions
Variables:• Population effect size (0, .1, .3, .5, .8)
• Mean sample size of study (20, 40, 80, 160)
• Number of studies in meta-analysis (5, 10, 15, 20, 25, 30)
• Variability in population = 0.16
Monte Carlo StudyMonte Carlo Study•• Effect sizes sampled from a population with a fixed Effect sizes sampled from a population with a fixed
or variable mean effect size.or variable mean effect size.
•• For each: 100,000 repetitionsFor each: 100,000 repetitions
Variables:Variables:•• Population effect size (0, .1, .3, .5, .8)Population effect size (0, .1, .3, .5, .8)
•• Mean sample size of study (20, 40, 80, 160)Mean sample size of study (20, 40, 80, 160)
•• Number of studies in metaNumber of studies in meta--analysis (5, 10, 15, 20, analysis (5, 10, 15, 20, 25, 30)25, 30)
•• Variability in population = 0.16Variability in population = 0.16
7-Mar-03 Andy Field Slide 24
Hedges Effect Size Estimates (Fixed)Hedges Effect Size Estimates (Fixed)ρ = 0.0
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.1
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.3
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.5
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s in
Meta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.8
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s in
Meta
-An
aly
sis
5
10
15
20
25
30
-0.010-0.005ρ+0.005+0.010
55
7-Mar-03 Andy Field Slide 25
HunterHunter--Schmidt Effect Size Estimates (Fixed)Schmidt Effect Size Estimates (Fixed)
ρ = 0.0
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s in
Meta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.1
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160S
tud
ies
in M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.3
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s in
Meta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.5
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s in
Meta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.8
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s in
Meta
-An
aly
sis
5
10
15
20
25
30
-0.010-0.005ρ+0.005+0.010
7-Mar-03 Andy Field Slide 26
Hedges Effect Size Estimates (Random)Hedges Effect Size Estimates (Random)ρ = 0.0
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.1
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.3
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.5
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s in
Meta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.8
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s in
Meta
-An
aly
sis
5
10
15
20
25
30
-0.20-0.15-0.10-0.05ρ+0.05+0.10+0.15
7-Mar-03 Andy Field Slide 27
HunterHunter--Schmidt Effect Size Estimates Schmidt Effect Size Estimates (Random)(Random)
ρ = 0.0
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.3
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.5
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.8
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
ρ = 0.0
Mean Sample Size
202020202020 404040404040 808080808080 160160160160160160
Stu
die
s i
n M
eta
-An
aly
sis
5
10
15
20
25
30
-0.20-0.15-0.10-0.05ρ+0.05+0.10+0.15
7-Mar-03 Andy Field Slide 28
0.050
0.075
0.100
0.125
0.150
0.175
202020202020404040404040808080808080
16016016016016016051015202530
51015202530
51015202530
51015202530
Typ
e I
Err
or
Rate
Sample Size
Studies in Meta-Analysis
Homogeneous Case
0.050
0.075
0.100
0.125
0.150
0.175
202020202020404040404040808080808080
160160160160160160
1015202530
1015202530
1015202530
1015202530
Typ
e I
Err
or
Rate
Sample Size
Studies in Meta-Analysis
Heterogeneous Case
Hedges & Colleagues
Hunter-Schmidt
Effect Sizes: Type I ErrorEffect Sizes: Type I Error
7-Mar-03 Andy Field Slide 29
Field (Submitted)Field (Submitted)Field (Submitted)Monte Carlo Study• Effect sizes sampled from a population with a
variable mean effect size.
• For each: 100,000 repetitions
Variables:• Population effect size (0, .1, .3, .5, .8)
• Mean sample size of study (20, 40, 80, 160)
• Number of studies in meta-analysis (5, 10, 20, 40, 80, 160)
• Variability in population effect sizes = 0.04, 0.08, 0.16, 0.32, 0.64
Monte Carlo StudyMonte Carlo Study•• Effect sizes sampled from a population with a Effect sizes sampled from a population with a
variable mean effect size.variable mean effect size.
•• For each: 100,000 repetitionsFor each: 100,000 repetitions
Variables:Variables:•• Population effect size (0, .1, .3, .5, .8)Population effect size (0, .1, .3, .5, .8)
•• Mean sample size of study (20, 40, 80, 160)Mean sample size of study (20, 40, 80, 160)
•• Number of studies in metaNumber of studies in meta--analysis (5, 10, 20, 40, analysis (5, 10, 20, 40, 80, 160)80, 160)
•• Variability in population effect sizes = 0.04, 0.08, Variability in population effect sizes = 0.04, 0.08, 0.16, 0.32, 0.640.16, 0.32, 0.64
7-Mar-03 Andy Field Slide 30
0.050
0.075
0.100
0.125
0.150
0.175
202020202020404040404040808080808080
160160160160160160510204080
160
510204080160
510204080160
510204080160
Type I
Err
or
Rate
Sample Size
Studies in Meta-Analysis
Population variance = .04
0.050
0.075
0.100
0.125
0.150
0.175
202020202020404040404040808080808080
160160160160160160510204080
160
510204080160
510204080160
510204080160
Typ
e I
Err
or
Rate
Sample Size
Studies in Meta-Analysis
Population variance = .08
Hedges & Colleagues
Hunter-Schmidt
Field (Submitted)Field (Submitted)Field (Submitted)
66
7-Mar-03 Andy Field Slide 31
0.050
0.075
0.100
0.125
0.150
0.175
202020202020404040404040808080808080
160160160160160160510204080
160
510204080160
510204080160
510204080160
Typ
e I
Err
or
Rate
Sample Size
Studies in Meta-Analysis
Population variance = .16
0.050
0.075
0.100
0.125
0.150
0.175
202020202020404040404040808080808080
160160160160160160510204080
160
510204080160
510204080160
510204080160
Type I
Err
or
Rate
Sample Size
Studies in Meta-Analysis
Population variance = .32
Hedges & Colleagues
Hunter-Schmidt
Field (Submitted)Field (Submitted)Field (Submitted)
7-Mar-03 Andy Field Slide 32
Field (Submitted)Field (Submitted)Field (Submitted)
0.050
0.075
0.100
0.125
0.150
0.175
202020202020404040404040808080808080
160160160160160160510204080
160
510204080160
510204080160
510204080160
Typ
e I
Err
or
Rate
Sample Size
Studies in Meta-Analysis
Population variance = .64
7-Mar-03 Andy Field Slide 33
ConclusionsConclusionsConclusionsCan meta-analysis be trusted?• Not if fixed effects methods are used when population
effect sizes vary (significance tests of most of the ones you read are likely to be type I errors)
• Hunter-Schmidt generally gives better estimates
• Hedges generally controls the Type I error better than H-S, but both methods OK when combining 80 or more studies
But…• Are we ever interested in significance tests anyway?
• Are moderator variables more interesting?
Can metaCan meta--analysis be trusted?analysis be trusted?•• Not if fixed effects methods are used when population Not if fixed effects methods are used when population
effect sizes vary (significance tests of most of the ones effect sizes vary (significance tests of most of the ones you read are likely to be type I errors) you read are likely to be type I errors)
•• HunterHunter--Schmidt generally gives better estimatesSchmidt generally gives better estimates
•• Hedges generally controls the Type I error better than Hedges generally controls the Type I error better than HH--S, but both methods OK when combining 80 or more S, but both methods OK when combining 80 or more studiesstudies
But…But…•• Are we ever interested in significance tests anyway?Are we ever interested in significance tests anyway?
•• Are moderator variables more interesting?Are moderator variables more interesting?