Dealing with small studies

Julian Higgins

School of Social and Community Medicine, University of Bristol, UK

Outline

• Theoretical problems with small studies: estimation of within-study variance

• Some empirical evidence relating to small studies

• Empirical problems of small studies: small study effects

• exclusion of small studies

• forest plots (including cumulative versions)

• funnel plots (including contour-enhanced versions)

• regression approaches

• alternative weighting approaches

• Some remarks on small meta-analyses

Theoretical problems with small studies: estimation of within-study variance

Acknowledgements: Mark Simmonds

Models for binary data (including logistic regression)

• All the methods in RevMan are two-stage methods

• obtain an estimate of treatment effect from each study

• compute a weighted average of these estimates

• An alternative is a one-stage method

• usually this requires individual participant data

• but we can create this for dichotomous outcomes

4

Alive Dead

Treatment rt1 ft1

Control rc1 fc1

Person Study Group Dead

1 1 0 0

2 1 0 ⁞

⁞ 1 ⁞ 0

⁞ 1 ⁞ 1

⁞ 1 ⁞ ⁞

⁞ 1 0 1

⁞ 1 1 0

⁞ 1 1 ⁞

⁞ 1 ⁞ 0

⁞ 1 ⁞ 1

⁞ 1 ⁞ ⁞

n1 1 1 1

5

IPD from a 2×2 table

rt1

ft1

fc1

rc1

Study 1 (n1 people)

Logistic regression

• With individual participant data, we can use the methods we use to analyse a single study

• except that we don’t have participant-level characteristics

• In particular we can use logistic regression

• but we can’t adjust for things like age, sex, severity

• A fixed-effect meta-analysis can be done using logistic regression, stratifying by study (a dummy variable for each study)

• A random-effects meta-analysis can be done using random-effects logistic regression (meqrlogit, previously xtmelogit)

• see Simmonds and Higgins, Statistical Methods in Medical Research (early view online)

6

Stata: meta-analysis using random-effects logistic regression

• With data in variables study, rt ,nt, rc, nc

reshape long n r, i(author) j(trt) string

gen treat = 0

replace treat=1 if trt=="t"

xi: meqrlogit s i.study i.treat || study: treat, nocons binomial(n)

• For out example (haloperidol) variables are author, rh, fh, rp, fp

gen nh = rh+fh

gen np = rp+fp

reshape long n r, i(author) j(trt) string

gen treat = 0

replace treat=1 if trt=="h"

xi: meqrlogit r i.author i.treat || author: treat, nocons binomial(n)7

Comparison of methods

Method for tauMethod for

confidence interval

Meta-analysis result

(95% CI)Estimate of τ2

Fixed-effect analysis OR = 2.85 (1.99, 4.10) n/a

DerSimonian-Laird Z (normal) OR = 4.20 (2.42, 7.30) 0.48

DerSimonian-Laird Hartung-Knapp (t) OR = 4.20 (2.31, 7.64) "

Paule-Mandel

(empirical Bayes)Z (normal) OR = 4.14 (2.40, 7.13) 0.45

Paule-Mandel

(empirical Bayes)Hartung-Knapp (t) OR = 4.14 (2.30, 7.45) "

REML Z (normal) OR = 4.38 (2.44, 7.86) 0.60

REML Hartung-Knapp (t) OR = 4.38 (2.33, 8.24)

Profile likelihood Z (normal) OR = 4.25 (2.39, 8.82) 0.51

Random-effects

logistic regressionOR = 4.72 (2.61, 8.53) 0.25

8

What about continuous data?

• For the individual study results, we should be using t-distributions for the confidence intervals

• But I don’t know of discussions of adjustment to weights in meta-analysis to account for small study issues

9

Empirical problems of small studies: small study effects

11

“Treatment effect estimates were significantly larger in smaller trials, regardless of sample size.”

12

13

14

16

17

18

“Systematic reviews of small trials increase waste by advertising to the scientific community inflated, often significant treatment effects that become smaller or absent when large, high-quality trials are done”

“To ignore results from small trials and postulate plausible treatment effects that would be clinically worthwhile would be preferable.”

Roberts and Kerr, Lancet 2015

Cumulative forest plots

.1 1 10

Cumulative MA (RE)

Marder

Arvanitis

Beasley

Bechelli

Vichaiya

Garry

Nishikawa ‘84

Chouinard

Reschke

Durost

Howard

Selman

Serafetinides

Borison

Spencer

Simpson

Nishikawa ‘82

Study

2.04 (0.51, 8.12)

128 1.51 (0.68, 3.35)

1.83 (0.83, 4.07)

1.13 (0.46, 2.78)

9.88 (1.97, 49.6)

27.34 (1.51, 496)

13.21 (0.72, 242)

5.76 (1.30, 25.5)

10.00 (1.79, 56.0)

19.25 (2.08, 178)

2.96 (0.60, 14.7)

9.71 (0.9, 103.0)

11.57 (0.56, 240)

9.21 (0.42, 201)

121.0 (6.7, 2188)

2.59 (0.11, 61.1)

3.32 (0.12, 91.6)

OR (95% CI)

101

81

59

58

50

47

43

40

34

30

29

27

24

24

23

20

Sample size

.01 .1 1 10 100

OR (95% CI)

Clinical trials of haloperidol for schizophrenia

Sequential considerations

• Cumulative meta-analyses are not an inferential tool

• Repeated testing → False positive findings

• Sequential methods are available to control type I error

• Indeed, sequential methods offer one way to address small studies

• Prevents early declarations of statistical significance based on the initial (potentially small) studies

20

Funnel plot

0.5

11

.52

2.5

Stan

dar

d e

rro

r

.01 .1 1 10Odds ratio (log scale)

0.5

11

.52

2.5

Stan

dar

d e

rro

r

.01 .1 1 10Odds ratio (log scale)

Funnel plot

0.5

11

.52

2.5

Stan

dar

d e

rro

r

.01 .1 1 10Odds ratio (log scale)

0.5

11

.52

2.5

Stan

dar

d e

rro

r

.01 .1 1 10Odds ratio (log scale)

0.5

1

Standarderror

0.1 1 10

Odds ratio

0

0.5

1

Standarderror

0.1 1 10

Odds ratio

0

0.5

1

Standarderror

0.1 1 10

Odds ratio

0

Subgroup 1

0.5

1

Standarderror

0.1 1 10

Odds ratio

0

Subgroup 2

0.5

1

Standarderror

0.1 1 10

Odds ratio

0

Subgroup 3

0.5

1

Standarderror

0.1 1 10

Odds ratio

0

Sterne et al (2011)

Contour-enhanced funnel plots

30

0.5

11

.52

Sta

nda

rd e

rro

r

0.02 0.05 0.1 0.2 0.5 1 2 5Odds Ratio (log scale)

0.1 > p > 0.05

0.05 > p> 0.01

p < 0.01

Studies

Peters et al (2008)

Contour-enhanced funnel plots

31

0.2

.4.6

.8

Sta

nda

rd e

rro

r

0.05 0.1 0.2 0.5 1 2 5Odds Ratio (log scale)

0.1 > p > 0.05

0.05 > p> 0.01

p < 0.01

Studies

10

.50

Stan

dar

d e

rro

r

0.1 1 10Odds ratio (log scale)

Extrapolating a funnel plot regression line

adjusted effect estimate

Moreno et al (2012)

Regression approach to small study effects

Magnesium trials

Risk ratio0.01 0.1 1 10

Study

Morton

Rasmussen

Smith

Abraham

Feldstedt

Shechter 1990

Ceremuzynski

LIMIT-2

Bertschat

Singh

Pereira

Golf

Thogersen

Shechter 1995

Estimates with 95% confidence intervals

ISIS-4

MAGIC

1.01 (0.97,1.07)Fixed effect

0.76 (0.62,0.92)Random effects

IV magnesium for acute MI (mortality)

(0.47,1.22)

Funnel plot: asymmetrical1

/ S

E(lo

g r

isk r

atio

)

Risk ratio

.1 .2 .5 1 2 5 10

0

10

20

30

40

Meta-analyses that are more robust to small study effects (1)

• Henmi and Copas (2010) propose to use the fixed-effect point estimate but with variance that acknowledges heterogeneity.

• Given

• then for any choice of ωi (constants), μ is unbiasedly estimated by

• with standard error

35

2

2

~ ,

~ ,

i i i

i

y N

N

ˆ i i

i

y

2 2 2

2ˆVar

i i

i

Meta-analyses that are more robust to small study effects (2)

• Choosing weights

• gives usual fixed-effect estimate, with variance

• We could naively plug in estimates of τ and σi

• Henmi and Copas derive a confidence interval that accounts for uncertainty in τ (see their paper for R code)

• Doi’s IVhet meta-analysis is the same, but he uses the naive plug-in variance

36

2 2 2

2ˆVar

i i

i

2

1i

i

2 2

4

2

2

ˆVar1

i

i

i

Some remarks on small meta-analyses

Small meta-analyses

• When heterogeneity is present, random-effects meta-analysis may be appropriate

BUT

• Estimation of heterogeneity is difficult in small meta-analyses

• A descriptive analysis of Cochrane systematic reviews found that 75% of meta-analyses contained 5 or fewer studies (Davey et al., 2011)

• A Bayesian approach is very useful in small meta-analyses:

• Allowance for all sources of uncertainty

• Incorporation of external evidence

38

Bayesian statistics

Prior

Data

Posterior

(Likelihood)

• We want to learn about some unknown quantities (e.g. odds ratio, mean difference, heterogeneity variance)

• A natural approach for accumulating data

Bayesian statistics

40

PosteriorLikelihoodPrior

The idea

• Analyse lots of previous meta-analyses and look at how much heterogeneity there was

• Produce off-the-shelf predictive distributions for different types of meta-analyses

• These can be used as prior distributions for heterogeneity variance in new meta-analyses

42

Examples of predictive distributions for 2

Outcome: all-

cause mortality;

Comparison:

Pharmacological

vs. placebo/ctrl.

Outcome:

subjective;

Comparison: Non-

pharma. (any

comparator)

0.1

.2.3

De

nsity

0.00001 0.0001 0.001 0.01 0.1 1 10 100

(a)

0

.05

.1.1

5.2

.25

De

nsity

0.00001 0.0001 0.001 0.01 0.1 1 10 100

(b)

Problem and solution

• Bayesian meta-analyses are computationally complex

• Usually done with simulation methods (Markov chain Monte Carlo) using WinBUGS or OpenBUGS

• An exciting recent development (Kirsty Rhodes et al, submitted) allows us to use informative prior distributions in Stata

• We make up some fake studies and analyse them alongside the real data

• We use the fake studies to learn about the heterogeneity variance (but they don’t contribute to treatment effect)

44

Fake studies to reflect some prior distributions for heterogeneity variance

45

Stata

• We’ll use the prior for subjective outcomes, pharmacol vs control

• 3 studies with lnOR = 0.346 and very small standard error

local new = _N+1

set obs `new’

replace author = "fake" in `new’

gen real=1

replace real=0 if author == "fake"

replace lnOR=0.346 if real == 0

replace se_lnOR=1E-10 if real == 0

expand 3 if real==0

metareg lnOR real, wsse(se_lnOR) reml z noconst eform

46

How many studies?Create new studyLabel as ‘fake’Dummy: real (1) vs fake (0)

Compute lnOR for fake... and very small SEMake it 3 fake studies

Regress on fake; no intercept

Results for haloperidol

Method for tauMethod for

confidence interval

Meta-analysis result

(95% CI)Estimate of τ2

Fixed-effect analysis OR = 2.85 (1.99, 4.10) n/a

DerSimonian-Laird Z (normal) OR = 4.20 (2.42, 7.30) 0.48

DerSimonian-Laird Hartung-Knapp (t) OR = 4.20 (2.31, 7.64) "

Paule-Mandel

(empirical Bayes)Z (normal) OR = 4.14 (2.40, 7.13) 0.45

Paule-Mandel

(empirical Bayes)Hartung-Knapp (t) OR = 4.14 (2.30, 7.45) "

REML Z (normal) OR = 4.38 (2.44, 7.86) 0.60

REML Hartung-Knapp (t) OR = 4.38 (2.33, 8.24) "

Profile likelihood Z (normal) OR = 4.25 (2.39, 8.82) 0.51

Random-effects

logistic regressionOR = 4.72 (2.61, 8.53) 0.25

Bayesian analysis

with priorOR = 3.79 (2.32, 6.18) 0.27

47

Concluding remarks

• Small studies pose problems

• They may have larger effect sizes (on average), which may be due to

• within-study bias

• reporting bias

• heterogeneity

• chance

• In principle it’s more important to focus on bias than to implement differential policies for smaller and larger studies

• although I recognize this is difficult in practice

• Random-effects logistic regression is available, and is probably the method we should always be using for binary data

48

References

• Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629-34.

• Henmi M, Copas JB. Confidence intervals for random effects meta-analysis and robustness to publication bias. Statistics in Medicine 2010; 29: 2969-83.

• Moreno SG, Sutton AJ, Thompson JR, Ades AE, Abrams KR, Cooper NJ. A generalized weighting regression-derived meta-analysis estimator robust to small-study effects and heterogeneity. Statistics in Medicine 2012; 31: 1407-17.

• Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Journal of Clinical Epidemiology 2008; 61: 991-6. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry.

• Simmonds MC, Higgins JPT. A general framework for the use of logistic regression models in meta-analysis. Statistical Methods in Medical Research. Published online 12 May 2014.

• Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, Carpenter J, Rücker G, Harbord RM, Schmid CH, Tetzlaff J, Deeks JJ, Peters J, Macaskill P, Schwarzer G, Duval S, Altman DG, Moher D, Higgins JPT. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011; 343: d4002

• Doi SA, Barendregt JJ, Khan S, Thalib L, Williams GM. Advances in the meta-analysis of heterogeneous clinical trials I: The inverse variance heterogeneity model. Contemporary Clinical Trials 2015; 45: 130-8.