Meta AnalysisStatistical analysis for

Systematic Reviews

of Randomised Clinical Trials

Contents

Contents- Example

- History – Fisher/Cochran, Peto, Cochrane, EBM

- Cochrane centre and Cochrane collaboration.

- Systematic reviews

- Statistical methodology- combining results- testing heterogeneity- comparability and quality of studies- publication bias

- Examples and Stata analysis

- Current issues – x-over trials, non RCTs, epidemiological studies

- Conclusions

blockers eg

Example:

Reports of several clinical trials of the effect of blockers on survival afterMyocardial Infarction are available (Yusuf et al looked at 65 trials of different types).

The main outcome variable is live/dead status at 1 month after treatment started.

The main test statistic is the Odds Ratio of surviving (treatment .v. placebo).

The statistical problems are:

- How to combine the ORs and obtain an overall OR with CLs

- How to assess and allow for the ‘true’ ORs varying with study (Heterogeneity)

- How to assess and allow for the quality and comparability of the studies

- How to assess whether these studies can be assumed a random sample of all such studies i.e. are some studies with different (?non-significant) results not being published – Publication Bias

History 1

History 2: Origins In the 1970s Richard Peto and others, because clinical trials of treatments for heart disease were often small and thus of low power, performed and published a number of systematic reviews of such trials.

Peto identified or devised appropriate methods for combining test statistics such as odds ratios, relative risks and differences between proportions and means from different studies.

In essence the problem is the same as that of multi-centre trials.

But variation in patient types, treatment regimes, outcome measures, study design and general trial procedures is much greater between unrelated studies than between centres following the same trial protocol.

This means that the aggregation of results has to be done very carefully and interpreted equally cautiously.

History 3

History 6: The Cochrane Collaboration

In 1992 Iain Chalmers at the Oxford Perinatal Research Unit and others set up the Cochrane Centre in Oxford to encourage and document systematic reviews of the effectiveness of Health interventions. It was named after a distinguished epidemiologist Archie Cochrane who in 1979 had written:"It is surely a great criticism of our profession (Medicine) that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.”

http://hiru.mcmaster.ca/cochrane/cochrane/archieco.htm

In October 1993 - at what was to become the first in a series of annual Cochrane Colloquia - 77 people from eleven countries co-founded 'The Cochrane Collaboration’ to encourage and document systematic reviews world-wide.

This happened in parallel with the ‘Evidence based medicine’ movement started by Sackett and others.

Systematic reviews

Proposers need to specify:

- the condition or disease

- the treatment comparisons of interest

- the types of study to be reviewed

- the strategy for searching the literature and elsewhere

- the outcome variables and planned analysis.

The Cochrane Collaboration publish guidelines entitled “ How to conduct a Cochrane review”

Stroke eg

Systematic Reviews

Results from 9 studies comparing Length of Stay (LOS) in hospital of Stroke patients receiving Specialist and Routine care

Specialist care Routine management Mean Mean

N LOS SD N LOS SD155 55 47 156 75 64 31 27 7 32 29 4 75 64 17 71 119 29 18 66 20 18 137 48 8 14 8 13 18 11 57 19 7 52 18 4 34 52 45 33 41 34110 21 16 183 31 27 60 30 27 52 23 20

paired plot

Stroke studies paired mean LOS

Mostly shorter LOS with specialist care,but some evidence of heterogeneity

Formulae

Paired t test structure

Combining Results The primary outcome in this study is the mean length of stay (LOS). So for study i we have: Study Treatment 1 Treatment 2 Differencei

No. Mean s.d. No. Mean s.d. 1 N11 y11 s11 N12 y12 s12

. .. .. .. .. .. ..

i Ni1 yi1 si1 Ni2 yi2 si2 yi = yi1 - yi2

. .. .. .. .. .. .. k Nk1 yk1 sk1 Nk2 yk2 sk2

pooled variance

spi 2= ((Ni1 -1) si1

2 + (Ni2 -1) si12)/(Ni1 + Ni2 -2)

var (yi) = si2 = spi

2 (1/ Ni1 + 1/ Ni2 ) Wi = 1/ si2

Combined estimate = Wi yi / Wi with variance = 1 / Wi

General Formulae

Formulae for combining test statisticsNormand, S-L. T (1999) Statist. Med 18, 321-359

Fixed-effects model: ~

Stroke results

M e t h o d P a r a m e t e r E s t i m a t o r V a r i a n c e

F i x e d - e f f e c t s m o d e l : iY ~ 2, isN

M L E

i i

i iiMLE W

YW ˆ 1 i iW

2/1 ii sW a s s u m e d k n o w n

B a y e s i a n

i i

i iiB

W

YW2

0

/~

12

0

i iW

2/1 ii sW , 20 a s s u m e d k n o w n

R a n d o m - e f f e c t s m o d e l 222 ,~,;,~ NsNY iiiiii

D e r S i m o n i a n a n d L a i r d 2

i

ii

wDL

w

ww

kQ2

2 1,0maxˆ N o n e p r o p o s e d

( M e t h o d o f m o m e n t s )

i DL

i iDLDL wi

Ywi

ˆ

ˆˆ 1ˆ DLi iw

22

2

ˆ

1ˆ,/1

DLiDLiii s

wsW

a s s u m e d k n o w n

Results from 9 studies comparing Length of Stay (LOS) in hospital of Stroke patients receiving Specialist and Routine care

Difference 95% CLs Weights Wi . bet. Means Low High Fixed Random

-20 -32.5 -7.5 0.0246 0.0036 -2 -4.8 0.8 0.4886 0.0042 -55 -62.7 -47.3 0.0654 0.0040 -71 -95.0 -47.0 0.0067 0.0026 -4 -12.8 4.8 0.0495 0.0039 1 -1.2 3.2 0.8173 0.0043 11 -8.1 30.1 0.0105 0.0030 -10 -15.6 -4.4 0.1245 0.0041 7 -1.9 15.9 0.0483 0.0039

Combined estimate MDw = Wi yi / Wi = -3.5

with variance = 1 / Wi = 0.6115 .............

Q = Wi (yi - MDw)2 = 241.1 ~ 28

Forest plot

The overall weighted mean differences are : Mean 95% CLs Fixed effects: -3.5 -5.0 -2.0 Random effects: -14.2 -24.8 -3.5 Test of heterogeneity: Q = 241.1

Treated as 2 with 8 df p = 0.0000 Funnel plot

Individual and combined results

Assessing publication bias

Meta analysis in RevMan

Results

Lidocaine treatment for MI v Placebo S-L T Normand Statist. Med (1999) 18, 321-359

2/39 5.1 1/43 2.34/44 9.1 4/44 9.1

6/107 5.6 4/110 3.67/103 6.8 5/100 5.07/110 6.4 3/106 2.8

11/154 7.1 4/146 2.7

% Death Rates

Lidocaine Control

Meta Analysis in RevMan

Dead Survived Dead Survived Odds Ratio Low High 12

p

2 37 1 42 2.27 0.20 26.07 0.01 0.93134 40 4 40 1.00 0.23 4.28 0.14 0.71086 101 4 106 1.57 0.43 5.74 0.14 0.71247 96 5 95 1.39 0.42 4.52 0.06 0.80667 103 3 103 2.33 0.59 9.27 0.83 0.3620

11 143 4 142 2.73 0.85 8.78 2.20 0.1378

95% CLsLidocaine Control

RevMan analysis of GBS data

RevMan forest plot

Review Manager (RevMan) from the Cochrane Collaboration

http://www.cochrane.org/software/revman.htm

New facilities in RevMan

New developments in RevMan

1. The generic inverse variance method;

2. The I2 measure of heterogeneity.

GIV theory

1. The generic inverse variance method

Traditionally each study needs to supply

• the mean, sd and n of the outcomes or the number with a particular outcome with the denominator n for each of the two treatment groups;

RevMan then calculates

• the mean difference (MD), odds ratio(OR) or relative risk (RR) as a comparison of the benefits of the two treatments;

• a variance for the MD, OR or RR from the sds or proportions from each treatment group;

• the square root of this variance as the standard error of the MD, OR or RR to obtain the confidence limits on the ‘forest’ plot;

• the inverse of this variance is then used to weight the MDs, ORs or RRs from the individual studies in the calculation of the combined estimate; data

Only need the first two columns… Difference 95% CLs Weights Wi bet. Means Var(diff) Low High Fixed Random -20 40.6 -32.5 -7.5 0.0246 0.0036 -2 2.1 -4.8 0.8 0.4886 0.0042 -55 15.3 -62.7 -47.3 0.0654 0.0040 -71 150.2 -95.0 -47.0 0.0067 0.0026 -4 20.2 -12.8 4.8 0.0495 0.0039 1 1.2 -1.2 3.2 0.8173 0.0043 11 95.4 -8.1 30.1 0.0105 0.0030 -10 8.0 -15.6 -4.4 0.1245 0.0041 7 20.7 -1.9 15.9 0.0483 0.0039 Combined estimate MDw = Wi yi / Wi = -3.5 -14.1

with variance = 1 / Wi = 0.6115 (-5.0,-2.0) (-24.5,-3.8)

Q = Wi (yi - MDw)2 = 241.1 ~ 28

I2 = 100*(Q – df)/Q = 96.7% forest plot

Trials in same order as table – GIV needs just 2 values for each trial

Compare with previous

9 Stroke trials:Fixed effect estimate = -3.49( -5.03 to -1.96)Random effect estimate = -14.62( -27.65 to -1.60)Q = 241.1; df = 8; p = 0.0000 I^2 = 96.7%

-10123456789

10

-100 -50 0 50

The overall weighted mean differences are : Mean 95% CLs Fixed effects: -3.5 -5.0 -2.0 Random effects: -14.2 -24.8 -3.5 Test of heterogeneity: Q = 241.1

Treated as 2 with 8 df p = 0.0000

Heterogeneity I2

Individual and combined results

2. The I2 measure of heterogeneity(1)

Heterogeneity – Systematic or Random

Systematic heterogeneity may arise because studies differ for a specific reason e.g. quality of study, biases or errors, type of patient, other non-random factors … usually implies that studies should be omitted or analysed separately;

Random heterogeneity between studies in the measures of the relative merits of two treatments is generally taken to represent differences that would occur in routine use of the treatments in similar institutions in the future. It is usually assumed to indicate that the combined estimate should be obtained using a random effects approach to take between study variation into account in the calculation of the combined estimate.

Stroke trials

Trials in same order as table – GIV needs just 2 values for each trial

outliers moved

9 Stroke trials:Fixed effect estimate = -3.49( -5.03 to -1.96)Random effect estimate = -14.62( -27.65 to -1.60)Q = 241.1; df = 8; p = 0.0000 I^2 = 96.7%

-10123456789

10

-100 -50 0 50

9 Stroke trials:Fixed effect estimate = -14.91( -16.45 to -13.38)Random effect estimate = -14.81( -17.04 to -12.58)Q = 10.6; df = 8; p = 0.2247 I^2 = 24.6%

-101234

56789

10

-100 -50 0 50

All mean differences about –15 … I2 = 24.6%

Current IssuesAreas

- Cross - over trials (treated as 2 parallel groups is safe)

- non randomised trials;

- observational clinical studies;

- epidemiological studies.

Methodology

- combining medians

- linear model and survival data analysis results (GIV)

- linear modelling approaches to combining study results;

(stratifying by study quality, patient and treatment variations etc)

- Bayesian methods

- Others....

Conclusions

Conclusions- Meta analysis of epidemiological studies is know to be fraught with many more problems than arise in the process of combining randomised clinical trials -- extra care is needed;

- must be aware that study results are data like any other and the problems are not different in principle to those faced in any statistical analysis;

- raw data from the studies is always better than summarised results;

- those doing a review should consult the Cochrane Guidelines.They need an exhaustive search strategy and careful classification of studies;

- The analysis needs to be cautious, although the particular method may not be crucial, and the form and nature of any heterogeneity found needs to be considered carefully in context and taken into account in the analysis.

References

Reference List 

1. Abrams K, Sanso B. Approximate Bayesian Inference for Random Effects Meta-analysis. Stats Med 1998;17:2:201-218.2. Aitkin M. Meta-analysis by Random Effect Modelling in Generalized Linear Models. Stats Med 1999;18:17/18:2343-2353.3. Austin H, Perkins LL, Martin DO. Estimating a Relative Risk Across Sparse Case-control and Follow-up Studies: A Method for Meta-analysis. Stats Med 1997;16:9:1005-1016.4. Begg CB, Pilote L. A model for incorporating historical controls into a meta-analysis. Biometrics 1991;47, No 3:899-906.5. Berkey CS, Hoaglin DC, Mosteller F, Colditz GA. A Random-effects Regression Model for Meta-analysis. Stats Med 1995;14:4:395-412.6. Berkey CS, Hoaglin DC, Antozak-Bouckoms A, Mosteller F, Colditz GA. Meta-analysis of Multiple Outcomes by Regression with Random Effects. Stats Med 1998;17:22:2537-2550.7. Berlin JA, Longnecker MP, Greenland S. Meta-analysis of Epidemiologic Dose-Response Data. Epidemiology 1993;4:218-228.8. Berry SM. Understanding and Testing for Heterogeneity Across 2 x 2 Tables: Application to Meta-Analysis. Stats Med 1998;17:20:2353-2370.9. Boutitie F, Gueyffier F, Pocock SJ, Boissel JP. Assessing Treatment-Time Interaction in Clinical Trials with Time to Event Data: A Meta-analysis of Hypertension Trials. Stats Med 1998;17:24:2883-2904.10. Brand R, Kragt H. Importance of Trends in the Interpretation of an Overall Odds Ratio in the Meta-analysis of Clinical Trials. Stats Med 1993;11 No 16:2077-2082.11. Brumback BA, Holmes LB, Ryan LM. Adverse Effects of Chorionic Villus Sampling: A Meta-analysis. Stats Med 1999;18:16:2163-2176.12. Buyse M, Molenberghs G, Burzykowski T, Renard D, Geys H. The validation of surrogate endpoints in meta-analysis of randomized experiments. Biostatistics 2000;1:1:49-68.13. Cole BF, Gelber RD, Goldhirsch A. A Quality-adjusted Survival Meta-analysis of Adjuvant Chemotherapy for Premenopausal Breast Cancer. Stats Med 1995;14:16:1771-1784.14. Cooper H, Hedges LV. The Handbook of Research Synthesis. Sage Publications, 1994;15. Copas J. What works?: selectivity models and meta-analysis. JRSS Series A 1999;162:1:95-110.16. Dear KBG. Iterative generalized least squares for meta-analysis of survival data at multiple times. Biometrics 1994;50:4:989-1003.17. Dersimonian R, Laird N. Meta-Analysis in Clinical Trials. Controlled Clinical Trials 1986;7:177-188.18. DuMouchel WH. Meta-Analysis. Stats Med 1995;14:5/6/7:679-686.19. Duval S, Tweedie R. Trim and Fill: A Simple Funnel-Plot-Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis. Biometrics 2000;56:2:455-463.20. Engels EA, Schmid CH, Terrin N, Olkin I, Lau J. Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses. Stats Med 2000;19:13:1707-1728.21. Fay MP, Graubard BI, Freedman LS, Midthune DN. Conditional logistic regression with sandwich estimators: application to a meta-analysis. Biometrics 1998;54:1:195-208.22. Follman DA, Proschan MA. Valid Inference in Random Effects Meta-Analysis. Biometrics 1999;55:3:732-737.23. Frost C, Clarke R, Beacon H. Use of Hierarchical Models for Meta-analysis: Experience in the Metabolic Ward Studies of Diet and Blood Cholesterol. Stats Med 1999;18:13:1657-1676.24. Geller NL, Proschan M. Meta-analysis of clinical trials; a consumers guide. J Biopharmaceutical Stats 1996;6:377-394.25. Goldstein H, Yang M, Omar R, Turner R, Thompson S. Meta-analysis using multilevel models with an application to the study of class size effects. JRSS Series C 2000;49:3:399-412.26. Greenland S. Quantitative Methods in the Review of Epidemiologic Literature. Epidemiologic Reviews 1987; 9:1-30.27. Hardy RJ, Thompson SG. A Likelihood approach to Meta-analysis with Random Effects. Stats Med 1996;15:6:619-630.

Reference List 

28. Hardy RJ, Thompson SG. Detecting and Describing Heterogeneity in Meta-analysis. Stats Med 1998;17:8:841-856.29. Hedges, Olkin. Statistical Methods for Meta-Analysis. Boston: Academic Press, 1985;30. Hutton JL, Williamson PR. Bias in meta-analysis due to outcome variable selection within studies. JRSS Series C 2000;49:3:359-370.31. Larose DT, Dey DK. Grouped Random Effects Models for Bayesian Meta-analysis. Stats Med 1997;16:16:1817-1830.32. Li Z. A Multiplicative Random Effects Model for Meta-Analysis with Application to Estimation of Admixture Component. Biometrics 1995;51:3:864-873.33. MacMahon S, Peto R, Cutler et al. Blood pressure, stroke and coronary heart disease, part 1, prolonged differences in blood pressure: prospective observational studies corrected for the regression dilution bias. Lancet 1990;335:765-774.34. Mathew T, Nordstrom K. On the Equivalence of Meta-Analysis Using Literature and Using Individual Patient Data. Biometrics 1999;55:4:1221-1223.35. Mosteller F, Colditz GA. Understanding Research Synthesis (Meta-Analysis). Annu.Rev.Public Health 1996;17:1-23.36. Normand SLT. Meta-analysis: Formulating, Evaluation, Combining, and Reporting. Stats Med 1999;18:3:321-360.37. Olkin I. Meta-Analysis: Reconciling the Results of Independent Studies. Stats Med 1995;14:5/6/7:457-472.38. Olkin I, Sampson A. Comparison of meta-analysis versus analysis of variance of individual patient data. Biometrics 1998;54:1:317-322.39. Pettiti D. Meta-analysis, Decision Analysis and Cost-effectiveness Analysis. Oxford: OUP, 1994;40. Platt RW, Leroux BG, Breslow N. Generalized Linear Mixed Models for Meta-analysis. Stats Med 1999;18:6:643-654.41. Rosenthal R. Meta-analytic procedures for social research. 1991;42. Schmid CH, Lau J, McIntosh MW, Cappelleri JC. An Empirical Study of the Effect of the Control rate as a Predictor of Treatment Efficacy in Meta-analysis of Clinical Trials. Stats Med 1998;17:17:1923-1943.43. Smith TC, Spiegelhalter DJ, Thomas A. Bayesian Approaches to Random-effects Meta-analysis: A Comparative Study. Stats Med 1995;14:24:2685-2700.44. Sullivan KM, Foster DA. Use of the Confidence Interval Function. Epidemiology 1990;1:1:39-42.45. Thompson SG, Smith TC, Sharp SJ. Investigating Underlying Risk as a Source of Heterogeneity in Meta-analysis. Stats Med 1997;16:23:2741-2758.46. Thompson SG, Pyke SDM, Hardy RJ. The design and analysis of paired cluster randomizede trials: an application of Meta-analysis Techniques. Stats Med 1997;16:18:2063-2080.47. Thompson SG, Sharp SJ. Explaining Heterogeneity in Meta-analysis: A Comparison of Methods. Stats Med 1999;18:20:2693-2708.48. Tritchler D. Modelling Study Quality in Meta-analysis. Stats Med 1999;18:16:2135-2146.49. Van Houwelingen HC. Meta-analysis; methods, limitations. Byocybernetics and Biomedical Engineering 1995;15:53-61.50. Walter SD. Variation in Baseline Risk as an Explanation of Heterogeneity in Meta-analysis. Stats Med 1997;16:24:2883-2900.51. Walter SD, Jadad AR. Meta-analysis of Screening Data: A Survey of the Literature. Stats Med 1999;18:24:3409-3424.52. Whitehead A, Whitehead J. A General Parametric Approach to the Meta-Analysis of Randomized Clinical Trials. Stats Med 1991;10:1665-1677.53. Whitehead A, Whitehead J. A general parametric approach to the meta-analysis of randomized clinical trials. Stats Med 1991;10:1655-1677.54. Whitehead A, Jones NMB. A Meta-analysis of Clinical Trials Involving Different Classifications of Response into Ordered Categories. Stats Med 1994;13:23/24:2503-2516.55. Whitehead A. A Prospectively Planned Cumulative Meta-analysis Applied to a Series of Concurrent Clinical Trials. Stats Med 1997;16:24:2901-2914.