Practical Meta-Analysis for the Social Sciences

Evan J. Ringquist

School of Public and Environmental Affairs

Indiana University

Workshop in Methods Presentation

January 11, 2013

Bloomington, IN 47405

1

2

“Meta-Analysis is not a fad. It is rooted in the fundamental values of the scientific enterprise: replicability, quantification, causal and correlational analysis. Valuable information is needlessly scattered in individual studies. The ability of social scientists to deliver generalizable answers to basic questions of policy is too serious a concern to allow us to treat research integration lightly. The potential benefits of meta-analysis method seem enormous.” (Bangert-Drowns 1986: 398)

Meta-Analysis Defined

• Meta-analysis is a systematic, quantitative, replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies.

• Meta-analysis, then, provides a powerful set of tools for aggregating knowledge.

3

Number of Articles Published Each Year in the “Meta-Analysis” Topic Category as Referenced by

the Social Sciences Citation Index

419801981198219831984198519861987198819891990199119921993199419951996199719981999200020012002200320042005200620072008200920102011

0

200

400

600

800

1000

1200

1400

artic

les p

er y

ear

Outline of Presentation

1. Motivation for Using Meta-Analysis in the Social Sciences

2. Quick Introduction to Meta-Analysis

3. Meta-Regression Models: The Basics

4. Meta-Regression Analysis for the Social Sciences– CRVE, GEE, and if we have time, HLM

5

Section I: Motivation

6

Cumulative Knowledge and the Scientific Enterprise

• A Central Goal of Science is the Accumulation and Aggregation of Knowledge. Yet the Social Sciences in General and Researchers in Public Management and Policy in Particular Have Found Cumulative Knowledge to be an Elusive Goal.

7

“Cumulative Knowledge” in Environmental Justice

• “Overwhelming evidence” of significant environmental inequities (Goldman 1993; see also Mohai and Bryant 1994)

• “even a reasonably generous reading of the empirical research alleging environment inequity . . . must leave room for profound skepticism” (Foreman 1998; see also Bowen 2001) 8

“Cumulative Knowledge” in Education Policy

• “Extensive research has been conducted on the academic success of students enrolled in school choice program nationwide. Rigorous studies show strong gains for voucher and scholarship tax credit recipients . . .” (Alliance for School Choice, 2010)

• “The reason that vouchers had subsided as a point of advocacy is because they don’t work.” (Randi Weingarten, President, American Federation of Teachers; Associated Press 2011).

•  “Few contemporary questions in American education have produced such wide-spread controversy as that regarding the potential of school vouchers to reduce inequality in student outcomes. . . these efforts notwithstanding, answers to the voucher question still appear uncertain.” (Cowen 2008)

9

Cumulative Knowledge in Public Management

• the more knowledge we create and diffuse, “the less we know, because our research is not aggregated or accumulated into substantial bodies of knowledge” (Van Slyke, O’Leary, and Kim 2010: 290.

• The first step in making Public Administration a stronger and more robust field is the need to aggregate knowledge in the sense of making it cumulative. (Rosenbloom 2010)

10

Cumulative Knowledge in Policy Studies

• “political scientists have spent (almost literally) countless articles and books proposing something like ‘laws’ or theories that, taken collectively . . . have produced infinitely more confusion than clarity” (DeLeon 1998: 150).

11

Cumulative Knowledge in Policy Evaluation

• Social scientists identify new and important questions having policy relevance. Initial studies examining these questions provide clear answers regarding, for example, the effectiveness of social policy interventions. Subsequent studies cast doubt on these initial conclusions, however, and with the proliferation of studies comes a proliferation of conclusions. The best efforts of social and behavioral researchers generate confusion and uncertainty rather than clarity. In the end, researchers conclude that the phenomena being studied are ‘hopelessly complex’ and move on to other questions. After several repetitions of this cycle, social and behavioral scientists themselves become cynical about their own work and express doubts about whether behavioral and social science in general is capable of generating cumulative knowledge or answers to socially important questions (Hunter and Schmidt 1996: 325-6).

12

Why Do Social Scientists Struggle with Knowledge Accumulation?

1. Epistemological Differences: Social Sciences ought not aim to generate generalizable “scientific” knowledge ala Chemistry.

2. Subject Matter Differences: Social scientists study organizations and institutions devised and populated by strategic actors, not molecules or organizms without agency.

3. Social Scientists Use The Wrong Tools for Aggregating Knowledge. 13

Section II: Introduction to Meta-Analysis

14

Meta-Analysis Defined (again)

• Meta-analysis is a systematic, quantitative, replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies.

• Meta-analysis, then, provides a powerful set of tools for aggregating knowledge.

15

The Language of Meta-Analysis

• Original Study: A piece of original research, published or unpublished, that aims to test a hypothesis and/or estimate a quantity of interest.

• Focal Predictor: The independent variable in an original study that measures the key exogenous concept associated with the research question of interest. 16

The Language of Meta-Analysis

• Effect Size: A standardized measure of the relationship between the focal predictor and the dependent variable in an original study. Effect sizes are the unit of analysis and the quantity of interest in a meta-analysis, often designated Θi. Without effect sizes there can be no meta-analysis.

17

The Language of Meta-Analysis

• Effect Size Variance: A measure of the uncertainty associated with a particular effect size.

• Fixed and Random Effects Models: Methods of weighting effect sizes that embody different assumptions about the effect size variance.

18

How Do We Calculate Effect Sizes?

• Three Families of Effect Sizes1. D-based effect sizes (standardized mean

differences, common in education and psychology)

2. Odds-based effect sizes (e.g. log odds of an event, common in medicine)

3. R-based effect sizes (partial correlation coefficients, unusual in the literature but most useful for social scientists) 19

Calculating R-based Effect Sizes from Original Studies

• r = √[t2 / (t2 + df)] [1]• r = √[Z2 / n] [2]• r = √[Χ2

1 / n] [3]

• V[r] = (1-r2)2 / (n-1) [4]

20

Fisher’s Corrections for R-based Effect Sizes and Variances

• Zr = 0.5 ln[(1+r) / (1-r)] [5]

• V[Zr] = 1/(n-3) [6]

21

Example: Are Pollution Emissions Higher in Black Neighborhoods?

• Ringquist 1997: t=3.06, df=29202– r=.02– Zr=.02– V[Zr] = .000034

• Downey 1998: t=3.38, df=112– r=.30– Zr=.31– V[Zr] = .0072

22

What Can We Do With Effect Sizes Statistically?

1. Communicate results from original research in a more meaningful fashion (not meta-analysis).

2. Combine (average) effect sizes to estimate the population effect size

3. Test null hypothesis that population effect size equals zero

4. Explain variation in effect sizes across original studies. We use “Meta-Regression” to account for this variation. 23

Combining Effect Sizes

• We calculate average effect sizes, or our estimate of the population effect size, by calculating a weighted average of all effect sizes from original studies where the weights are inverse variances

• Θbar= ΣwiΘi / Σwi [7]

• Wi = 1/vi24

Combining Effect Sizes: Fixed Effects Models

• Fixed Effects models assume that effect sizes vary across original studies only due to sampling error:– Θi = Θ + ei [8], so

– E[Θi]= Θ [9], and

– ei ~ N(0,vi) [10]

25

Combining Effect Sizes: Fixed Effects Models

• Fixed effects meta-regression assumes that effect sizes conditional upon moderator variables differ only due to sample size

• Fixed effects estimates apply only to sample of original studies in hand

26

Combining Effect Sizes: Random Effects Models

• Random Effects models assume that effect sizes are normally distributed random variables– Θ ~ N(µΘ, τ2) [11]

– Θi = µΘ + ei [12]

– E[Θi] = µΘ [13], and

– ei ~ N(0,vi + τ2) [14]

– Always use random effects in the social sciences!

27

Estimating the Random Effects Variance Component

• Using Restricted Maximum Likelihood– Le(τ2) = -.5 * Σ [ln(vi + τ2) + ((ei

2 / (vi + τ2) - .5 * ln |X’v-1X| [15]

• Using Method of Moments (MOM)– tr(M) = Σvi

-1 – tr [(Σvi-1 XX’)-1 (Σvi

-2 XX’)] [16]

• Using MOM Approximation– τ2 = [SSEols / (m-k-1)] – vbar [17]

28

What Can We Do With Effect Sizes Substantively?

• Estimate severity of problems– E.g., environmental inequities– E.g., effects of climate change

• Measure important quantities of interest– E.g., statistical value of a life– E.g., hedonic pricing of environmental

amenities

29

What Can We Do With Effect Sizes Substantively?

• Program Evaluation– E.g., effectiveness of educational vouchers– E.g., effectiveness of job training programs

• Theory Testing and Development– E.g., Top-down vs. bottom-up implementation– E.g., effects of negative campaign ads– E.g., Ricardian equivalence

30

Section III: Introduction to Meta-Regression

31

“exploration of the pattern of variation in effect sizes among studies is a far more important goal of meta-analysis than the

construction of powerful tests of null hypotheses.” (Osenberg et al. 1999: 1105)

32

Moving Meta-Analysis to the Social Sciences

• Techniques for calculating and combining effect sizes developed for synthesizing the results from experiments. Can these same techniques really be used to synthesize results from the non-experimental multivariate models common in the social sciences?

33

Three Critiques of Meta-Analysis in the Social Sciences

1. Parameter estimates from the general linear model are not comparable.

2. Effect sizes cannot be calculated from multivariate models.

3. Parameter estimates from different models estimate different population parameters, and therefore cannot be combined. 34

Response to Critique #1:

• While regression parameters are not comparable, meta-analysis does not combine parameter estimates. Rather, meta-analysis combines effect sizes which are standardized with respect to scale. This is not a valid critique.

35

Response to Critique #2

• In fact, the same formulas that can be used to calculate r-based effect sizes from experimental studies can be used to calculate partial correlations from regression models, probit models, etc. (see Greene 1993: 180)

36

Response to Critique #2• Example 1: Panel regression model,

Y=continuous, K=33, N=307,538– Pcorr = .0099, Zr = .0096

• Example 2: Probit model, Y=dichotomous, K=8, N=14131– Pcorr = .1081, Zr = .1062

• Example 3: OLS regression model, Y=continuous, K=24, N=285– Pcorr = .1619, Zr = .1565

37

Response to Critique #3

• E.g., Study 1– Y = b0 + b1X1 + b2X2 + b3X3 + e

• E.g., Study 2– Y = b0 + b1X1 + b2X2 + b4X4 + e

• E[b11] ≠ E[b12], β11 ≠ β12, and Θ1 ≠ Θ2

• Therefore dependent variable Θi measures fundamentally different quantities

38

Response to Critique #3• Critique #3 is a valid critique, and this is

why calculating average effect sizes is of little value in the social sciences – these average effect sizes combine estimates of different population parameters, and therefore are not valid estimates of any useful quantity of interest.

• Thankfully, we can address Critique #3 using Meta-Regression. 39

What Do We Need to Conduct a Meta-Regression?

• Measure of effect size that is comparable across original studies employing different measures, models, and samples

• Measure of (un)certainty or effect size variance

• Moderator variables that account for variability in effect sizes

40

Introduction to Meta-Regression

• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)– Θi is the effect size

– X represents moderator variables accounting for differences in effect sizes within and across studies• Scientifically Interesting Moderators (Rubin 1992)

– e.g., differences attributable to target characteristics– e.g., differences attributable to program or policy design

• Scientifically Uninteresting Moderators– e.g., differences attributable to estimation technique or

research design 41

Number of Publications Using Phrase “Meta-Regression” in Google Scholar, 1990-2011

42

Example: Synthesizing Research on Educational Vouchers

• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)– Θi is the estimated point biserial correlation (effect

size Zr) between the use of an educational voucher and student standardized test scores from a particular statistical model in a particular original study.

43

Example: Educational Vouchers

• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)– X1 moderator variable identifying effect sizes from

models limiting sample to black children– X2 moderator variable identifying effect sizes from

voucher programs available to religious schools– X3 moderator variable identifying effect sizes from

models not using random assignment to treatment and control groups.

44

Example: Educational Vouchers

• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)– b0 average effect of vouchers on student test scores

for all students in experimental studies where vouchers use is limited to secular schools

– b1 differential effect of vouchers for black students

– b2 differential effect from voucher programs that include religious schools

– b3 differential voucher effect from quasi-experimental studies

45

46

Independent Variables

CRVE Parameter Estimates (St.Err.)

CRVE 95% CI

GEE Parameter Estimates (St.Err.)

GEE 95% CI

Black Student 0.0296** 0.0030 0.0562 0.0370*** 0.0143 0.0598(0.0130) (0.0116)

Hispanic Student -0.0470*** -0.0680 -0.0259 -0.0413*** -0.0577 -0.0248(0.0103) (0.0084)

Religious Schools -0.0312 -0.0693 0.0069 0.0180 -0.0574 0.0213(0.0187) (0.0201)

Matched Control Groups -0.0097 -0.0226 0.0033 -0.0069* -0.0148 0.0009(0.0064) (0.0040)

Convenience Control Groups -0.0073 -0.0246 0.0101 -0.0082 -0.0243 0.0078(0.0085) (0.0082)

Pretest Included -0.0326*** -0.0560 -0.0092 -0.0228* -0.0464 0.0008(0.0115) (0.0120)

Intercept (Baseline Effect Size) 0.1097*** 0.0573 0.1620 0.0830*** 0.0243 0.1416(0.0257) (0.0299)

Adjusted R2a

Psuedo-R2a

I2a

Sample Size

.44

.68

25.01%

611

    na

na

na

611

   

Obtaining Estimates for Meta-Regression Models

• OLS Estimator of Meta-Regression Model– E[b] = β = (X’X)-1 X’Θ [18]– var[b] = σ2(X’X)-1 [19]

• But OLS is incorrect for two reasons:1. Θi is heteroskedastic (recall assumptions)

2. OLS weights all observations equally (unequal variances means unequal certainty) 47

Generalized Least Squares in Meta-Regression

• In OLS, ee’ = σ2I • In GLS, ee’ = σ2Ω

– where Omega is a diagonal matrix with proportional error variances on main diagonal

• GLS estimates then are:– E[bgls] = βgls = (X’Ω-1X)-1 X’Ω-1Θ [20]

– var[bgls] = σ2 (X’Ω-1X)-1 [21]48

Weighted Least Squares in Meta-Regression

• Find a weight matrix W so that W’W = Ω-1

• Then the WLS meta-regression estimates become:– E[bwls] = βwls = (X’W’WX)-1 X’W’WΘ [22]

– var[bwls] = σ2 (X’W’WX)-1 [23]

49

Weighted Least Squares in Meta-Regression (cont.)

• The good news is that we have a handy estimate of the diagonal elements of the weight matrix W: the effect size variances (or more properly, their square roots)

• The bad news is that fixed effects variance estimates are almost certainly wrong, so we need to use random effect variances. 50

RE Variance Component τ2

• Cannot include moderator variables for all independent variables or all design elements in original studies

• Random Effects variance component τ2 can be conceived of as reflecting the impact of unobserved variables on Θi

51

Weights in Random Effects Meta-Regression

(v1 + τ2) 0 0 0

0 (v2 + τ2) 0 0

ee’remr = σ2 . . . .

0 0 0 (vn + τ2)

• With weights as the square roots of the diagonal elements of this matrix. 52

Estimating the Random Effects Meta-Regression Model in Practice• Option 1: Use the WLS function in Stata,

SAS, or some other statistical package.– Don’t do this. All WLS algorithms treat weights

as proportional variance weights rather than as absolute variance weights. This means that meta-regression parameter standard errors will be incorrect.

53

Estimating the Random Effects Meta-Regression Model in Practice• Option 2: Use the VWLS command in

Stata.– This will work as long as you import the

proper random effects variance weights (actually, their standard deviations). These must be calculated outside of the VWLS model. This works because VWLS treats the imported variances as known and absolute rather than as estimated and proportional. 54

Estimating the Random Effects Meta-Regression Model in Practice• Option 3: Use the “Metareg” Command in

Stata– While Stata does not include commands for

meta-analysis, the Stata User Group (SUG) has produced a number of good meta-analysis ADO files, including a random effects meta-regression algorithm. These commands can be downloaded from the Stata website.

55

Estimating the Random Effects Meta-Regression Model in Practice• Option 4: Use “Manually Weighted” Least

Squares– Estimate the REMR model by first manually

weighting each observation by the square root of the proper random effects variance, then using OLS on the transformed data. While this is more work than using the VWLS or Metareg commands, it is necessary when estimating more advanced meta-regression models. 56

Comparison of REMR Results in Environmental Justice (n=680)

Metareg VWLS MWLS

(MLE) (MOM) (MLE)Intercept .068 .073 .068

(.006) (.008) (.006)

Pollution -.020 -.020 -.020

(.008) (.011) (.008)

Census Tracts .011 .009 .011

(.006) (.008) (.006)

Select on DV -.078 -.093 -.078

(.019) (.025) (.019) 57

Consequences of Inadequately Addressing Effect Size Heterogeneity

• Inadequately addressing heterogeneity in effect sizes means you have a misspecified REMR model.

• Consequences if r(Me,Mk) = 0?

• Consequences if r(Me,Mk) ≠ 0?

• Consequences of remaining effect size heterogeneity in REMR model are akin to specification error in OLS regression

13

Validity Check: Can Meta-Regression Really Recover the

Quantity of Interest from Original Studies?

59

Simulation Strategy• Two-level Monte-Carlo simulation

– Assume Population Regression Model Y=β0 + β1 X1 + β2 X2 + β3 X3 + ε.

• First level simulate 10,800 original studies by• Generate X from distribution N(0,1,Cxx), generate

Y using β1 = .2, .7 (X1 is focal predictor), generate ε ~ N (0, 30)

• Allow measurement of X and Y to vary across studies [X, X*1000, and ln(X)]

• Estimate sample regression models that employ combinations of X1, X2, and X3

60

Simulation Strategy (cont.)• Second level simulate 1000 meta-analyses of

sample size 500• Estimate meta-regression model

– Y=β0 + β1 M1 + β2 M2 + ε

– M represent moderator variables

• Key Question: Can REMR recover partial correlation parameter from population regression model in original studies?

61

Simulation Results

62

Table 7.10. Ability of REMR Model Recover Population Parameter of Interest in the Presence of Effect Size Heterogeneity and Variation in the Measurement of X and Y in Original Studies Corr

(Xj,Xk)

Population Parameter of Interest

Estimate of b0

Parameter Estimate

for Moderator Variable X2

Parameter Estimate

for Moderator Variable X3

Predicted Value of

Population Parameter of Interest

.60

.3234

.5008

-.1648

-.0253

.3107

.60

.0950

.1644

-.0552

-.0065

.1027

Section IV: Advanced Meta-Regression

63

“Meta-Analytic techniques developed to date have not addressed the problem of

combining information on second-order effects, such as partial regression

coefficients. Consequently, meta-analysis has not been very useful in the synthesis of evidence from causal/explanatory research."

(Becker 1992: 342)

64

Remaining Problems in Meta-Regression

1. Non-Independence of Observations

2. Unequal Numbers of Effect Sizes per Original Study

65

Problem 1:Non-Independence of Observations• Reasons for Correlated Observations

1. Common data sets across studies

2. Common research teams across studies

3. Single original study produces multiple effect sizes (generally incorporates both 1 and 2)

• Consequences of correlated observations for REMR: incorrect parameter st. errors 66

Addressing Problem 1:The Approach in Meta-Analysis

• Traditional Solutions in Meta-Analysis1. Select one effect size per study to use in

meta-analysis

2. Calculate average effect size per study and use this in meta-analysis

67

Addressing Problem 1:The Approach in Meta-Analysis

• Why the Traditional Solutions are Really Bad Ideas:1. Assumes each additional effect size from a

particular study contains no information

2. Cannot estimate effect of moderators within studies, only across studies

3. Dramatically reduces sample size and statistical power

68

Addressing Problem 1:The Approach in Econometrics

• The Traditional Solution in Econometrics: Cluster Robust Variance Estimation (CRVE)– i.e., Huber-White Standard Errors as adjusted

for clusters by Liang and Zeger (1986) and Arellano (1987).

– Effect sizes Θig clustered by original study G

69

Addressing Problem 1:The Approach in Econometrics

• In Matrix Form:

V[bREMRg] = (M’W’WM)-1 (Σ Mg’W’(egeg’)WMg) (M’W’WM)-1 [24]

70

71

    (τ2+v11+σ21)

ρ1 0 . . . . . . . . 0

    ρ1 (τ2+v21+σ21)

.                 .

    0 . σ22** ρ2 ρ2 ρ2           .

    .   ρ2 σ22** ρ2 ρ2           .

    .   ρ2 ρ2 σ22** ρ2           .

    .   ρ2 ρ2 ρ2 σ22**           .

    .           .         .

ee’ = .             .       .

    .               .     0

    .                 σ2g** ρg ρg

    .                 ρg σ2g** ρg

    0 . . . . . .   0 ρg ρg σ2g**

Addressing Problem 1 Using Example REMR Model

CRVE

REMR REMRIntercept .068 .068

(.006) (.011)

Pollution -.020 -.020

(.008) (.015)

Census Tracts .011 .011

(.006) (.010)

Select on DV -.078 -.078

(.019) (.032) 72

Addressing Problem 1 Using CRVE

• CRVE REMR model should be standard practice in the Social Sciences -- BUT

• Only three examples of published CRVE CRVE models. Why?

• Researchers developing statistics of meta-analysis unfamiliar with developments in econometrics, and vice versa.

73

Extensions to the CRVE REMR Model

1. Small number of clusters? – Use the “Wild Cluster Bootstrap”– Cameron, Gelbach, and Miller 2008

2. Two-way clustering? E.g., effect sizes clustered by study and by data set?– Use “multi-level clustering”– Cameron, Gelbach, and Miller (2011)

74

Problem 2: Unequal Numbers of Effect Sizes Per Study

• Reasons for Unequal Numbers of Effect Sizes Per Study– Varying numbers of subgroup analyses– Varying sensitivity tests

• Consequences of Unequal Numbers of Effect Sizes Per Study– Few studies may dominate data set and

REMR results, threatening external validity75

Addressing Problem 2:The Approach in Meta-Analysis

• Traditional Solutions in Meta-Analysis1. Select one effect size per study to use in

meta-analysis

2. Calculate average effect size per study and use this in meta-analysis

• As we have seen, this is really bad advice

76

Addressing Problem 2:The Approach in Econometrics

• The Econometric approach to Problem 3 generally takes one of two forms.1. Relative “importance weights” attached to

observations (e.g., weight = 1/g)

2. Generalized Estimating Equations (GEE)

77

Addressing Problem 2 Using GEE

• A Simple Introduction to GEE– E[Θi] = h (Mb) [25]

• Where h is a link function (identity in regression)

– V[Θi|Mi] = σ2i = g([Mbi] / φ [26]

• i.e., the variance is a function of the expectation• φ (phi) is a scale parameter that we will ignore

today

78

Addressing Problem 2 Using GEE

• A Simple Introduction to GEE (cont.)– Assume we have three effect sizes from two

studies: [Θ11 , Θ21 , Θ22] • Θ = (1/3) Θ11 + (1/3) Θ21 + (1/3) Θ22 [27]

• Θ = (1/2) Θ11 + (1/2) Θ21 [28]

• Θ = (1/2) Θ11 + (1/4) Θ21 + (1/4) Θ22 [29]

– All are unbiased estimates of population effect size Θ, but with different variances

79

Addressing Problem 2 Using GEE

• A Simple Introduction to GEE (cont.)– GEE estimates Θ using an optimal weighting

scheme employing the correlation among observations from the same cluster (ρ). In this simple example:

Θ = [((1+ρ) / (3+ρ)) Θ11] + [(1 / (3+ρ)) Θ21] + [(1 / (3+ρ)) Θ22][30]

80

Addressing Problem 2 Using GEE

• A Simple Introduction to GEE (cont.)– We still need an estimate of the conditional

variance of the sample effect size Θi. GEE estimates this variance (σ2) using:• σ2 = (A)1/2 R (A)1/2 [31]• A has E[Θi ] = h Mb on the diagonal

• R is the “working correlation matrix”• For REMR, the exchangeable working correlation

matrix is most appropriate 81

Addressing Problem 2 Using GEE

• Obtaining GEE Estimates• Estimate a naïve model for Θ assuming independent

observations to obtain starting values for b and e.• Use e to estimate the elements of the working correlation

matrix R.• Re-estimate the structural model with weights derived

from the estimated values of ρ in R.• Iterate steps 2 and 3 to convergence.

82

Addressing Problem 2 Using GEE

A Bonus from Using GEE to Estimate the REMR Model: If we use the “empirical” matrix ee’ rather than the “model” matrix ee’, GEE also controls for Problem 2

– GEE calculates parameter estimates controlling for clustering

– Empirical GEE calculates parameter variance estimates controlling for clustering

83

Addressing Problem 4 Using Example REMR Model

CRVE GEE

REMR REMR REMRIntercept .068 .068 .067

(.006) (.011) (.011)

Pollution -.020 -.020 -.015

(.008) (.015) (.014)

Census Tracts .011 .011 .008

(.006) (.010) (.009)

Select on DV -.078 -.078 -.072

(.019) (.032) (.021) 84

Addressing Problem 2 Using GEE

• GEE is an excellent choice for estimating the REMR model.

• GEE REMR is quite rare – fewer than half a dozen in the published literature

• Preferred over CRVE when number of effect sizes are highly unequal across studies. CRVE preferred in balanced samples. 85

Estimating Advanced REMR Models in Stata

• Stata cannot estimate directly either the CRVE REMR or the GEE REMR model.

• Neither VWLS nor Metareg can estimate these models.

• Models must be estimated using MWLS data in conjunction with “robust cluster” or “xtgee” commands in Stata.

86

The Take Aways

1. Meta-regression is a powerful and under-utilized tool for synthesizing research in the social sciences.

2. When estimating meta-regression models, social scientists should use the CRVE REMR or the GEE REMR model.

3. Social scientists should not use the standard REMR model or the HLM REMR model.

87

88

“(Meta-analysis) is going to revolutionize how the sciences . . . handle data. And it is going to be the way many arguments will end.”

T. Chalmers, quoted in Mann 1990: 480

Evan Ringquisteringqui@indiana.edu

89