1

DrPH Seminar Session 3

Quantitative Synthesis – Focusing on Heterogeneity

Qualitative Synthesis

– e.g., GRADE

Mei Chung, PhD, MPH Research Assistant Professor

Nutrition/Infection Unit, Department of Public Health and Community Medicine, Tufts School of Medicine

FORMULATE STUDY QUESTIONESTABLISH PROTOCOL

CRITICAL APPRAISAL PAPER SELECTION per PROTOCOL

DATA EXTRACTION and QUALITY ASSESSMENT

ANALYSIS and INTERPRETATIONWEIGHTED AVERAGE

REGRESSIONSENSITIVITY ANALYSIS

LITERATURE SEARCH / RETRIEVAL

Steps of Performing a Systematic Review

Ask

Identify

Acquire

Appraise

Synthesize

1

2

Meta-analysis 101

n  The term “meta-analysis” typically referred to meta-analysis of study-level, summary data

n  The term “pooled analysis” typically referred to meta-analysis of individual-level data across different studies

n  Conventional meta-analysis methods and software are designed for combining RCTs (comparisons of ES between two independent groups)

2

Reasons for Meta-analysis

n  Improve the power to detect a small difference if the individual studies are small

n  Improve the precision of the effect measure n  Compare the efficacy of alternative interventions

and assess consistency of effects across study and patient characteristics

n  Gain insights into statistical heterogeneity n  Help to understand controversy arising from

conflicting studies or generate new hypotheses to explain these conflicts

n  Force rigorous assessment of the data

3

3

Principles of Combining Data for Basic Meta-analyses

n  For each analysis, one study should contribute only one treatment effect.

n  The effect estimate may be for a single outcome or a composite.

n  The outcome being combined should be the same (or similar-based on clinical plausibility) across studies.

n  Know your question. The question drives your study selection, synthesis, and interpretation of the results.

4

Things to Know about the Data before Combining

n  Biological and clinical plausibility n  Scale of effect measure n  Studies with small number of events do

not give reliable estimates

5

4

True Associations May Disappear When You Combine Data Inappropriately

6

Apparent Association May Be Seen When There is None

7

5

Changes in the Same Scale May Have Different Meaning

Effe

ct o

f int

eres

t

Variable of interest

A

B

C

DBoth A–B and C–D involve a change of one absolute unit. A–B change (1 to 2) represents a 100% relative change. C–D change (7 to 8) represents only a 14% relative change.

8

Commonly Encountered Comparative Effect Measures

Type of Data Corresponding Effect Measure

Continuous Mean difference (e.g., mmol, mmHg) Standardize mean difference (effect size) Correlation

Dichotomous Odds ratio, risk ratio, risk difference

Time to event Hazard ratio

9

6

What Is the “Average (Overall)” Treatment-Control BP Difference?

10

Study N Mean difference

mmHg

95% CI

A 554 -6.2 -6.9 to -5.5

B 304 -7.7 -10.2 to -5.2

C 39 -0.1 -6.5 to 6.3

Simple Average

(−6.2) + (−7.7) + (−0.1) 3

= −4.7 mmHg

n

xX

k

ii∑

== 1

11

Study N Mean difference

mmHg

95% CI

A 554 -6.2 -6.9 to -5.5

B 304 -7.7 -10.2 to -5.2

C 39 -0.1 -6.5 to 6.3

7

Weighted Average

(554 x −6.2) + (304 x −7.7) + (39 x −0.1) 554 + 304 + 39

= −6.4 mmHg

∑

∑

=

== k

ii

k

iii

w

xwX

1

1

12

Study N Mean difference

mmHg

95% CI

A 554 -6.2 -6.9 to -5.5

B 304 -7.7 -10.2 to -5.2

C 39 -0.1 -6.5 to 6.3

General Formula: Weighted Average Effect Size (d+)

∑

∑

=

=+ = k

ii

k

iii

w

dwd

1

1

where: di = effect size of the ith study wi = weight of the ith study k = number of studies

13

8

Calculation of Weights

n  Generally the inverse of the variance of treatment effect (that captures both study size and precision)

n  Different formula for odds ratio, risk ratio, risk difference

n  Readily available in books and software

14

Heterogeneity (Diversity)

n  Is it reasonable (are studies characteristics and effects sufficiently similar) to estimate an average effect?

n  Types of heterogeneity –  Clinical diversity –  Methodological diversity –  Statistical heterogeneity

15

9

Clinical Diversity

Are the studies of similar treatments, populations, settings, design, etc., such that an average effect would be clinically meaningful?

16

Methodological Diversity

Are the studies of similar design and conduct such that an average effect would be meaningful?

17 17

10

Statistical Heterogeneity

n  Is the observed variability of effects greater than that expected by chance alone?

n  Two statistical measures are commonly used to assess statistical heterogeneity – Cochran’s Q-statistics –  I2 index

18

Cochran’s Q-Statistics χ2 (chi-square) Test for Homogeneity

( )21

2)1( +

=− −== ∑ ddwQ ii

k

idfkχ

di = effect measure d+ = weighted average

19

Q-statistics measures between study variation

19

11

20

I2 Index and its Interpretation

•  I2 describes the percentage of total variation in study estimates that is due to heterogeneity rather than chance.

•  The value of I2 index ranges from 0% to 100%. •  A value of 25% is considered as low heterogeneity, 50% as

moderate, and 75% as large. •  The I2 index is independent of the number of studies in the

meta-analysis. It could be compared directly between meta-analyses.

2 max ,11

QHk⎛ ⎞= ⎜ ⎟−⎝ ⎠

22

2

1HIH−

=

Higgins JPT, et al. Measuring inconsistency in meta-analyses. BMJ 2003;327:557–560.

21 A Meta-analysis with a Large Degree of Heterogeneity

12

22

Choi et al. Developmental Fluoride Neurotoxicity: A Systematic Review and Meta-Analysis

Heterogeneity

n Diversity of studies in a meta-analysis n Typically abundant n Arguably the most important role of meta-

analytic methodologies is to quantify, explore, and explain between-study heterogeneity

13

Statistical heterogeneity

Statistical heterogeneity exists when the results of the individual studies are not “consistent” among themselves

Clinical heterogeneity

Methodological heterogeneity

Biases

Chance

Statistical heterogeneity

Clinical vs. statistical heterogeneity

n Clinical and methodological heterogeneity is abundant. Our aim is to explore it, and use these observations to formulate interesting hypotheses.

n Often, but not always, clinical and methodological heterogeneity will result in a statistically significant test

n Chance, technical issues or biases may result in statistically significant results in heterogeneity tests

14

HETEROGENEOUS TREATMENT EFFECTS

IGNORE INCORPORATEESTIMATE(insensitive) EXPLAIN

FIXED EFFECT MODEL

DO NOT COMBINE WHEN

HETEROGENEITY IS PRESENT

RANDOM EFFECTS MODEL

SUBGROUP ANALYSES

META-REGRESSION(control rate, covariates)

Dealing with Heterogeneity 26

Lau J, et al. Quantitative synthesis in systematic review. Ann Intern Med 1997; 127:826.

Fixed- vs. Random-Effects Meta-analysis

n  Fixed Effect Model (FEM): Assumes a common treatment effect.

n  Random Effect

Model (REM): In contrast with the FEM, the REM accounts for between study variation.

27

15

Weights of the Fixed Effect and Random Effects Models

*1*

vvw

ii +=

ii vw 1=

Random Effects Weight Fixed Effect Weight

where: vi = within study variance

v* = between study variance

28

Commonly used Statistical Methods for Combining 2x2 Tables

Odds Ratio

Risk Ratio

Risk Difference

Fixed Effect Model

Mantel-Haenszel Peto Exact

Inverse variance weighted

Mantel-Haenszel

Inverse variance weighted

Inverse variance weighted

Random Effects Model

DerSimonian& Laird DerSimonian& Laird

DerSimonian& Laird

29

16

30 Figure 1. An example of a fixed-effects MA

Figure ㄉ. An example of a random-effects MA

Borenstein et al. A basic introduction to fixed-effect and random-effects models for meta-analysis Res. Syn. Meth. 2010, 1 97--111

Dealing with HeterogeneityHETEROGENEOUS

TREATMENT EFFECTS

IGNORE INCORPORATEESTIMATE(insensitive) EXPLAIN

FIXEDEFFECTS MODEL

DO NOT COMBINEWHEN

HETEROGENEITYIS PRESENT

RANDOMEFFECTSMODEL

SUBGROUPANALYSES

META-REGRESSION(control rate,covariates)

17

Trea

tmen

t effe

ct

Trea

tmen

t effe

ct

variable of interest

META-REGRESSION modeling summary data

OVERALL ESTIMATE combining summary data

RESPONSE SURFACE modeling individual patient data

varia

ble 2

Trea

tmen

t effe

ct

variable 1Tr

eatm

ent e

ffect

SUBGROUP ANALYSES differentiating effects in subgroups

Summary: Statistical Models of Combining 2x2 Tables

n  Most meta-analyses of clinical trials combine treatment effects (risk ratio, odds ratio, risk difference) across studies to produce a common estimate, using either a fixed effect or random effects model.

n  In practice, the results using these two models are similar when there is little or no heterogeneity.

n  When heterogeneity is present, the random effects model generally produces a more conservative result (smaller Z-score) with a similar estimate but with a wider confidence interval. However, there are rare exceptions of extreme heterogeneity where random effects model may yield counterintuitive results.

33

18

Caveats

n  Many assumptions are made in meta-analyses, care is needed in the conduct and interpretation.

n  Most meta-analyses are retrospective exercises, suffering from all the problems of being an observational design.

n  We cannot make up missing information or fix poorly collected, analyzed, or reported data.

34 34

Summary n  Basic meta-analyses can be easily carried out with

one of many readily available statistical software. n  Relative measures are more likely to be

homogeneous across studies and is generally preferred.

n  Random effects model is the appropriate statistical model in most instances.

n  Decision to do a meta-analysis should be based on a well-formulated question, appreciation of the heterogeneity of the data, and understanding of how the results will be used.

35

19

Q & A

36

Qualitative Synthesis

n  Also called, Grading Strength of Evidence –  Distinct from rating quality of articles/

studies –  Many tools, e.g. GRADE

n  Qualitative synthesis is required for a systematic review –  Meta-analysis is optional

37

20

GRADE

WHO GRADE process http://cebgrade.mcmaster.ca/ WHO guidelines and GRADE: An overview summary (60 minutes) http://cebgrade.mcmaster.ca/Summary/index.html

38

Dividing Homework

n  Session 4: Mock expert panel –  Jefferson T et al. Vaccines for preventing

influenza in healthy adults. Cochrane Database Syst Rev. 2010 Jul 7;(7):CD001269. doi: 10.1002/14651858.CD001269.pub4. Review. Update in: Cochrane Database Syst Rev. 2014;3:CD001269. PubMed PMID: 20614424. l  Three outcomes : Three students (what a

coincidence!)

39