A Quarter Century on, Where are we?

Tom Stanley, Hendrix College

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

(MRA) is at once a framework in which to organize and interpret exact and inexact replications, to review more objectively the literature and explain its disparities, and to engage in the self-analysis of investigating the socioeconomic phenomenon of social scientific research itself– Stanley & Jarrell (1989, p. 168).

Stanley, T.D. and S.B. Jarrell (1989) Meta-regression analysis: A quantitative method of literature surveys. Journal of Economic Surveys, 3: 161-70

Exponential @

18%/year

Where are we going?

Has MRA’s promise been realized?

More objective reviews of economic research Explanation of disparate research findings¿ Investigation of the socio-economics of

economics research?

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Investigation of the socio-economics of economics research?

• Female economists find less wage discrimination against women than do male researchers (S&J, 1998; J&S, 2004, Weichselbaumer & Winter-Ebmer, 2005)

• Publication bias is the result of professional incentives and the pressure to publish.

• Researcher ideology affects reported results (Doucouliagos and Paldam, 2006).

• The Research Cycle: Reported findings generally confirm a novel hypothesis; later, the incentive for rejection increases (S, J & D, 2008).

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Has MRA’s promise been realized?

More objective reviews of economic research Explanation of disparate research findings Investigation of the socio-economics of

economics research, . . . but much more to do.¿ Framework to organize and interpret exact and

inexact replications?

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Framework to organize and interpret exact and inexact replications?

• Deakin University’s (Chris Doucouliagos) meta-data repository.

• Bob Reed, Maren Duvendack, and others have expressed interest in organizing something more formal.

• Piers Steel and metaBUS.• Needless to say, . . . .

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

How about S&J’s (1989) exact MRA model? Does anyone still use it?

bi = b + SbkZik +ei (3)

Where: “bi is the reported estimate of b from the ith study. . . Zik the meta-independent variable which measures relevant characteristics of an empirical study and explains its systematic variation” (p. 164)

• Zik might include:

1. Dummy variables for omitted variables.

2. Specification variables

3. Sample Size

4. Author characteristics

5. Data characteristics

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Due to obvious Heteroskedasticity, S&J 89 recommended the WLS version of (3):

ti= bi/Sbi = b (1/Sbi

) + Sbk (Zik/Sbi) +ei /Sbi

(4)

• t-value, ti, is the dependent variable and

precision (1/Sbi) is an independent variable.

• WLS is neither fixed- nor random-effects, in practical application, WLS is better than both (Stanley and Doucouliagos, 2013a&b, Deakin SWP).

• We never wanted to use fixed- or random-effects, in spite of citing Hedges and Olkin (1985).

• Had we included the intercept, we would have fully anticipated current practice {FAT-PET-MRA}.

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Neither Fixed

• Fixed-effect MRA: same as our WLS regression, but divides SEs by square root of MSE (H&O, 85)• causes SEs & CIs to be too small & too narrow. • assumes that policy makers wish to make

inferences to a population that is identical, in every respect, to past research. Like that happens!

• So why divide by root MSE???• WLS already correct the SEs for both excess

heterogeneity and heteroskedasticity• Fixed-effect MRA is never relevant in economics.

T.D.

Sta

nley

, He

n d

rix C

olle

ge

MAE

R-N

et S

epte

mbe

r 12,

201

4

Nor Random

• Random-effects MRA adds a second error term, ni , to the conventional meta-regression model,

bi = b + SbkZik +ni + ei (1)

Where ni is assumed to be normal and independent of the sampling errors, ei, and the moderators, Zik.

• Problems:• In economics, excess heterogeneity is systematic!

• Typically, ni will be the result of omitting relevant variables; thus, it introduces bias.

• In the 1980s, we saw no reason to use FE or RE• Now, we know that we should never use FE or RE!

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Problems and Issues—2014

Multiple Meta-Regression (MRA)• Should we divide meta-regression SEs by the

square root of MSE? No! (Hedges and Olkin, 1985)

• Does Random-Effects MRA become biased with publication bias? YES! (Stanley and Doucouliagos, 2013b)

Meta-analysis (MA)—Weighted Averages• Fixed-Effects Estimator (FEE): confidence intervals

are too narrow if there is heterogeneity. • Random-Effects Estimator (REE): can be very

biased with publication bias {already widely established}

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Neither Fixed nor Random 4 2014

• In two recent simulation papers, Chris and I show:• WLS is as good as Fixed or Random-Effects

under the best conditions for FE and RE.• If we are making inferences to policy settings or

to future research, WLS is much better than FE.• When there is publication bias, WLS is much

better than RE.• Worse still: all tests of either heterogeneity or

publication bias are known have low power.• Therefore, in practice, we should never use

either FE or RE. . . . Never!

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Our two working papers:

Stanley, T.D., Doucouliagos H(C). 2013a. Neither Fixed nor Random: Weighted least squares meta-regression analysis. SWP, Economics Series 2013-1, Deakin University. http://www.deakin.edu.au/buslaw/aef/workingpapers/papers/2013_1.pdf

Stanley, T.D., Doucouliagos H(C). 2013b. Better than Random: Weighted least squares meta-regression analysis. SWP, Economics Series 2013-2, Deakin University. http://www.deakin.edu.au/buslaw/aef/workingpapers/papers/2013_2.pdf

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Weighted Least Squares (WLS-MRA)

• MRAs should never be estimated by OLS, because there is much variation among the reported SEs of bi or effecti

• Enter WLS: =(MtW-1M)-1MtW-1b (2)

Where:

=

β

2

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

The Gauss-Markov Theorem{Proportional Heterogeneity Invariance}

• As long as W in (2) is known up to some unknown proportion, s2, WLS { } is the Best {Minimum Variance} Linear Unbiased Est.

• Invariance to proportional excess heterogeneity is a robustness property of the Gauss-Markov Theorem and WLS.

• It is not an assumption.

β

T.D.

Sta

nley

, Hen

drix

Col

lege

SRS

M

July

2, 2

014

Traditional, Unrestricted WLS

replaces with:

= (5)

and is estimated by the WLS residuals, automatically

2S

2

22

21

00

00

00

LSE

SE

SE

2S

T.D.

Sta

nley

, Hen

drix

Col

lege

SRS

M

July

2, 2

014

Simulation Design• Generate Yj and estimate b from:

Yj = 100 + b X1j +a2 X2j +a3i X3j + ej (6)

• Half the studies omit the relevant variable X2i

• a3i ~N(0, ) adds excess random heterogeneity by always omitting relevant variable X3i

• When b = 1, r between Y and X1 is .27

• n= {62, 125, 250, 500, 1000} in primary regressions

• X1j , X2j, X3j are generated randomly, but X2j & X3j are forced to be correlated with X1j .

• Fixed- or random-effects model is always true.

2h

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Simulation Design—Cont. • Experiment 1: 10,000 WLS, RE and FE-MRAs are

calculated with one moderator variable, Mi = {0,1}, reflecting whether the original study omitted X2i, or not, (Tables 1 & 2)

bi = b0 + b1 Mi + ui (7)

Where: bi is the ith primary study’s estimate of b

• Experiment 2: Experiment 1 plus 50% of the studies select statistically significant results, and either MRA (7), above, or a multiple FAT-PET-MRA is used. (Tables 3 & 4)

bi = b0 + b1 Mi + b2 SEi + ui (8)

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Table 1: Coverage PercentagesMRA

n h True

Effect I2 FE-MRA RE-MRA WLS-

MRA 20 0 0 .0948 .9489 .9544 .9505 20 0.125 0 .2433 .8769 .9218 .9350 20 0.25 0 .6014 .7067 .9082 .9079 20 0.5 0 .8503 .4740 .9191 .9000 20 1.0 0 .9465 .3088 .9254 .9110 20 2.0 0 .9761 .2277 .9265 .9339 20 4.0 0 .9858 .1909 .9233 .9464 80 0 0 .0936 .9495 .9553 .9525 80 0.125 0 .2469 .8741 .9429 .9350 80 0.25 0 .6011 .7007 .9371 .9058 80 0.5 0 .8493 .4769 .9495 .9079 80 1.0 0 .9465 .3173 .9433 .9167 80 2.0 0 .9761 .2384 .9460 .9440 80 4.0 0 .9858 .2047 .9472 .9528 20 0 1 .0593 .9545 .9603 .9531 20 0.125 1 .3186 .8738 .9187 .9278 20 0.25 1 .6465 .7070 .8996 .9064 20 0.5 1 .8687 .4688 .9183 .8996 20 1.0 1 .9517 .3125 .9220 .9119 20 2.0 1 .9777 .2301 .9227 .9378 20 4.0 1 .9863 .1851 .9252 .9455 80 0 1 .0589 .9532 .9568 .9532 80 0.125 1 .3179 .8704 .9382 .9282 80 0.25 1 .6471 .7040 .9444 .9138 80 0.5 1 .8683 .4765 .9460 .9049 80 1.0 1 .9517 .3153 .9427 .9240 80 2.0 1 .9777 .2364 .9468 .9393 80 4.0 1 .9863 .1947 .9436 .9566

Average .5349 .9352 .9286

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Table 2: Bias and MSEMRA n h True

Effect I2 RE-MRA

|Bias| WLS-MRA

|Bias| RE-MRA

MSE WLS-MRA

MSE 20 0 0 .0948 .00059 .00041 .00554 .00549 20 0.125 0 .2433 .00105 .00124 .00829 .00845 20 0.25 0 .6014 .00091 .00157 .01498 .01687 20 0.5 0 .8503 .00085 .00031 .03555 .04661 20 1.0 0 .9465 .00087 .00282 .11340 .13435 20 2.0 0 .9761 .00157 .00014 .40591 .35193 20 4.0 0 .9858 .01341 .00148 1.6279 .88102 80 0 0 .0936 .00048 .00051 .00110 .00109 80 0.125 0 .2469 .00059 .00040 .00173 .00179 80 0.25 0 .6011 .00029 .00021 .00331 .00386 80 0.5 0 .8493 .00030 .00077 .00833 .01066 80 1.0 0 .9465 .00023 .00031 .02669 .02919 80 2.0 0 .9761 .00012 .00099 .09887 .06928 80 4.0 0 .9858 .00240 .00203 .38644 .15535 20 0 1 .0593 .00046 .00042 .00564 .00558 20 0.125 1 .3186 .00186 .00172 .00825 .00837 20 0.25 1 .6465 .00147 .00164 .01487 .01706 20 0.5 1 .8687 .00068 .00107 .03607 .04736 20 1.0 1 .9517 .00118 .00358 .11352 .13372 20 2.0 1 .9777 .00075 .00247 .39659 .33989 20 4.0 1 .9863 .01035 .01111 1.6164 .83945 80 0 1 .0589 .00067 .00067 .00110 .00109 80 0.125 1 .3179 .00013 .00013 .00172 .00177 80 0.25 1 .6471 .00068 .00060 .00333 .00389 80 0.5 1 .8683 .00009 .00048 .00822 .01035 80 1.0 1 .9517 .00012 .00063 .02720 .02953 80 2.0 1 .9777 .00163 .00005 .09808 .06986 80 4.0 1 .9863 .00195 .00040 .38633 .15414

Average .00163 .00136 .19483 .12064

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Simulation Results I:“A house divided by itself cannot stand”—A. Lincoln

• FE-MRA: unacceptable SEs in most actual applications. Thus, Do Not Divide by root MSE!• When there is no heterogeneity and FE-MRA is

true, WLS-MRA is equivalent to FE-MRA.

• RE-MRA: When RE-MRA’s model is true, WLS-MRA provides acceptable and comparable coverage, and its bias and MSE are a bit better.• Irony: WLS-MRA dominates RE-MRA in those exact

circumstances for which RE-MRA is designed—large additive, excess random heterogeneity

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Table 3: Bias and MSE with 50% Publication Bias MRA

n h True

Effect I2 RE-MRA

|Bias| WLS-MRA

|Bias| RE-MRA

MSE WLS-MRA

MSE 20 0 0 .1689 .0348 .0328 .0151 .0147 20 0.125 0 .3241 .0581 .0510 .0218 .0209 20 0.25 0 .5697 .1140 .0957 .0414 .0397 20 0.5 0 .8102 .2367 .1964 .1084 .1035 20 1.0 0 .9264 .4510 .3470 .3259 .2677 20 2.0 0 .9670 .8138 .5692 1.0391 .6824 20 4.0 0 .9809 1.5212 .8595 3.6524 1.6393 80 0 0 .1551 .0361 .0345 .0039 .0037 80 0.125 0 .3589 .0668 .0593 .0085 .0075 80 0.25 0 .6184 .1322 .1148 .0237 .0200 80 0.5 0 .8372 .2659 .2250 .0824 .0643 80 1.0 0 .9362 .4900 .3891 .2687 .1815 80 2.0 0 .9701 .8939 .6092 .8868 .4365 80 4.0 0 .9818 1.6566 .8880 3.0617 .9305 20 0 1 .0825 .0135 .0128 .0056 .0056 20 0.125 1 .2358 .0168 .0129 .0083 .0085 20 0.25 1 .5325 .0350 .0221 .0155 .0171 20 0.5 1 .8083 .0916 .0583 .0412 .0477 20 1.0 1 .9255 .2415 .1669 .1567 .1483 20 2.0 1 .9666 .5566 .3541 .6540 .4317 20 4.0 1 .9806 1.2326 .6554 2.8299 1.1672 80 0 1 .0450 .0101 .0096 .0012 .0012 80 0.125 1 .2591 .0158 .0115 .0020 .0019 80 0.25 1 .5926 .0314 .0172 .0042 .0041 80 0.5 1 .8364 .0940 .0570 .0163 .0133 80 1.0 1 .9349 .2564 .1740 .0886 .0559 80 2.0 1 .9695 .6142 .3756 .4553 .1978 80 4.0 1 .9817 1.3571 .6591 2.1485 .5624

Average .4049 .2521 .5703 .2527

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Table 4: Bias, MSE (50% Pub’bias) FAT-PET-MRA MRA

n h True

Effect I2 RE-MRA

|Bias| WLS-MRA

|Bias| RE-MRA

MSE WLS-MRA

MSE 20 0 0 .0948 .16575 .16364 .05218 .05157 20 0.125 0 .2433 .15454 .14634 .06086 .05960 20 0.25 0 .6014 .10523 .08881 .07024 .07462 20 0.5 0 .8503 .01049 .00049 .11731 .14752 20 1.0 0 .9465 .08977 .07918 .32996 .35647 20 2.0 0 .9761 .11570 .06047 1.0056 .84226 20 4.0 0 .9858 .19965 .03766 3.0241 1.8469 80 0 0 .0936 .14279 .14135 .02492 .02454 80 0.125 0 .2469 .12187 .11082 .02212 .01996 80 0.25 0 .6011 .06794 .04757 .01648 .01601 80 0.5 0 .8493 .04014 .05133 .02536 .03112 80 1.0 0 .9465 .14829 .13834 .08918 .08054 80 2.0 0 .9761 .21051 .13721 .26245 .14630 80 4.0 0 .9858 .38580 .06001 .84128 .25204 20 0 1 .0593 .02702 .02664 .01722 .01720 20 0.125 1 .3186 .03110 .02868 .02553 .02607 20 0.25 1 .6465 .03162 .02711 .04590 .05264 20 0.5 1 .8687 .02446 .02144 .10850 .13508 20 1.0 1 .9517 .01027 .00171 .30811 .33235 20 2.0 1 .9777 .01052 .04896 .97668 .75452 20 4.0 1 .9863 .05311 .16689 2.8764 1.6098 80 0 1 .0589 .02505 .02471 .00391 .00390 80 0.125 1 .3179 .02933 .02690 .00602 .00611 80 0.25 1 .6471 .03398 .02921 .01039 .01168 80 0.5 1 .8683 .02473 .02216 .02287 .02753 80 1.0 1 .9517 .01618 .00175 .06726 .06377 80 2.0 1 .9777 .04481 .03268 .22621 .12240 80 4.0 1 .9863 .21000 .11280 .70017 .23085

Average .09038 .06553 .40490 .26226

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Simulation Results IIWhen there is publication bias:

• WLS-MRA dominates RE-MRA. • WLS-MRA always has less bias and, on

average, substantially lower MSE

• Irony: WLS dominates RE in those exact circumstances for which RE-MRA is designed.

• When RE is better, it is not much better, and those cases cannot be identified, in practice.

• Thus, there is No Reason to use Random-Effects Meta-Regression. . . . . . Ever!

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Why WLS works so well when there is high heterogeneity?

For very large heterogeneity, the random heterogeneity term, , will dominate sampling error, , and its variation, making the overall variance of the estimate, , roughly proportional to .

2j

2j2j

2j

0

1

2

3

4

5

6

7

8

9

SE

-squ

ared

0 50 100 150 200 250

Heterogeneity Variance

Unlike S & J, S & J’s MRA Model

Still works after all these Years

Multiple MRA

• Should we divide MRA SEs by √MSE? Never!

• Is RE-MRA biased with publication bias? Yes!• WLS-MRA dominates RE-MRA with or

without Correcting for Publication Bias. • WLS also dominates REE and FEE

weighted averages when combining Cohen’s d from RCTs.

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

The Task Ahead

• We need more realistic simulation studies• Alternative modeling strategies {general-

to-specific; Bayesian modeling}• Unbalanced panel MRA models.

• We need to continue to raise the quality of MRA applications, making them more robust, comprehensive and rigorous.

• Wish us luck at ASSA in Boston.

T.D.

Sta

nley

, Hen

drix

Col

lege

M

AER-

Net

Sep

tem

ber 1

2, 2

014

Thank You!