META-ANALYSES: APPROPRIATE GROWTH META-ANALYSES: APPROPRIATE GROWTH OR MALIGNANT TUMOR? OR MALIGNANT TUMOR?

Ian ShrierIan Shrier MD, PhD, Dip Sport Med (FACSM)MD, PhD, Dip Sport Med (FACSM)

• Associate Professor, Dep’t of Fam Med, McGill UniversityAssociate Professor, Dep’t of Fam Med, McGill University• Centre for Clinical Epidemiology and Community Studies, Centre for Clinical Epidemiology and Community Studies,

SMBD-Jewish General Hospital and McGill UniversitySMBD-Jewish General Hospital and McGill University• Past-president, Canadian Academy of Sport MedicinePast-president, Canadian Academy of Sport Medicine

Canadian Academy of Sport MedicineL’Académie Canadienne de Médecine du Sport

GROWTH VS QUALITY?

• Cochrane Collaboration – 10 pointsCochrane Collaboration – 10 points collaborationcollaboration building on the enthusiasm of individualsbuilding on the enthusiasm of individuals avoiding duplicationavoiding duplication minimizing biasminimizing bias keeping up to date keeping up to date striving for relevance striving for relevance promoting access promoting access ensuring qualityensuring quality continuity continuity enabling wide participationenabling wide participation

• Cochrane Collaboration – 10 pointsCochrane Collaboration – 10 points collaborationcollaboration building on the enthusiasm of individualsbuilding on the enthusiasm of individuals avoiding duplicationavoiding duplication minimizing biasminimizing bias keeping up to date keeping up to date striving for relevance striving for relevance promoting access promoting access ensuring qualityensuring quality continuity continuity enabling wide participationenabling wide participation

GrowthGrowthGrowthGrowth

GROWTH VS QUALITY?

• Cochrane Collaboration – 10 pointsCochrane Collaboration – 10 points collaborationcollaboration building on the enthusiasm of individualsbuilding on the enthusiasm of individuals avoiding duplicationavoiding duplication minimizing biasminimizing bias keeping up to date keeping up to date striving for relevance striving for relevance promoting access promoting access ensuring qualityensuring quality continuity continuity enabling wide participationenabling wide participation

• Cochrane Collaboration – 10 pointsCochrane Collaboration – 10 points collaborationcollaboration building on the enthusiasm of individualsbuilding on the enthusiasm of individuals avoiding duplicationavoiding duplication minimizing biasminimizing bias keeping up to date keeping up to date striving for relevance striving for relevance promoting access promoting access ensuring qualityensuring quality continuity continuity enabling wide participationenabling wide participation

GrowthGrowthGrowthGrowth

QualityQualityQualityQuality

XX XX

DATA ENTRY

OBJECTIVE?

A Meta-analysis is an Observational Study, A Meta-analysis is an Observational Study,

Subject to Selection Bias, Confounding, Subject to Selection Bias, Confounding,

and Measurement Error Like Any Other!and Measurement Error Like Any Other!

A Meta-analysis is an Observational Study, A Meta-analysis is an Observational Study,

Subject to Selection Bias, Confounding, Subject to Selection Bias, Confounding,

and Measurement Error Like Any Other!and Measurement Error Like Any Other!

Platt, personal communicationPlatt, personal communication

Computer programs and standardized scoring make it Computer programs and standardized scoring make it very easy to make mistakes that are difficult for the very easy to make mistakes that are difficult for the reviewer to pick up.reviewer to pick up.

Computer programs and standardized scoring make it Computer programs and standardized scoring make it very easy to make mistakes that are difficult for the very easy to make mistakes that are difficult for the reviewer to pick up.reviewer to pick up.

OVERVIEW

• Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction

• AnalysesAnalyses

• Interpretation of EvidenceInterpretation of Evidence

• Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction

• AnalysesAnalyses

• Interpretation of EvidenceInterpretation of Evidence

INCLUSION CRITERIA - SEARCHINCLUSION CRITERIA - SEARCH

• Search StrategySearch Strategy ““Does Vitamin C prevent death due to cancer in the Does Vitamin C prevent death due to cancer in the

general population”general population”

• Search StrategySearch Strategy ““Does Vitamin C prevent death due to cancer in the Does Vitamin C prevent death due to cancer in the

general population”general population”

33,30033,300Vitamin CVitamin C#1#1

HitsHitsTermsTermsSearchSearch

34,41534,415Ascorbic AcidAscorbic Acid#2#2

9,5459,545AscorbateAscorbate#3#3

4,3764,376#3 NOT #2#3 NOT #2#4#4

42,09842,098#1 OR #2 OR #3#1 OR #2 OR #3#5#5

INCLUSION CRITERIA – RCTs?

Good (56%)

Poor (44%)

INCLUSION CRITERIA – RCTs?

Furlan, 2006Furlan, 2006Furlan, 2006Furlan, 2006

DATA ABSTRACTION

• Peer-review for stretching article:Peer-review for stretching article:

““The RR 0.79 … was based on the article in Pope The RR 0.79 … was based on the article in Pope 2000.”2000.”

• Peer-review for stretching article:Peer-review for stretching article:

““The RR 0.79 … was based on the article in Pope The RR 0.79 … was based on the article in Pope 2000.”2000.”

… … multivariate analysis HR=1.04 (0.82,1.33)multivariate analysis HR=1.04 (0.82,1.33) … … multivariate analysis HR=1.04 (0.82,1.33)multivariate analysis HR=1.04 (0.82,1.33)

• Actual articleActual article

The X article … Cox regression … differing The X article … Cox regression … differing exposure times for different subjects. …univariate exposure times for different subjects. …univariate hazard ratio of 0.95 (CI: 0.77,1.18)hazard ratio of 0.95 (CI: 0.77,1.18)

• Actual articleActual article

The X article … Cox regression … differing The X article … Cox regression … differing exposure times for different subjects. …univariate exposure times for different subjects. …univariate hazard ratio of 0.95 (CI: 0.77,1.18)hazard ratio of 0.95 (CI: 0.77,1.18)

OVERVIEW

• Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction

• AnalysesAnalyses

• Interpretation of EvidenceInterpretation of Evidence

• Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction

• AnalysesAnalyses

• Interpretation of EvidenceInterpretation of Evidence

ANALYSESANALYSES

• Effect ModifiersEffect Modifiers Study-level confoundersStudy-level confounders Subject-level confoundersSubject-level confounders

• Publication BiasPublication Bias Papers with significant results are more Papers with significant results are more

likely to be published (i.e. selected)likely to be published (i.e. selected)

• Reporting BiasReporting Bias Outcomes that are statistically significant Outcomes that are statistically significant

are more likely to be reported in publications are more likely to be reported in publications (i.e. selected)(i.e. selected)

• Effect ModifiersEffect Modifiers Study-level confoundersStudy-level confounders Subject-level confoundersSubject-level confounders

• Publication BiasPublication Bias Papers with significant results are more Papers with significant results are more

likely to be published (i.e. selected)likely to be published (i.e. selected)

• Reporting BiasReporting Bias Outcomes that are statistically significant Outcomes that are statistically significant

are more likely to be reported in publications are more likely to be reported in publications (i.e. selected)(i.e. selected)

ANALYSESANALYSES

Fixed vs RandomFixed vs RandomFixed vs RandomFixed vs Random

Effect EstimateEffect EstimateRR vs ORRR vs OR

Effect EstimateEffect EstimateRR vs ORRR vs OR

ANALYSESANALYSES

Effect EstimateEffect EstimateRR vs RDRR vs RD

Effect EstimateEffect EstimateRR vs RDRR vs RD

OVERVIEW

• Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction

• AnalysesAnalyses

• Interpretation of EvidenceInterpretation of Evidence

• Inclusion Criteria / Data abstractionInclusion Criteria / Data abstraction

• AnalysesAnalyses

• Interpretation of EvidenceInterpretation of Evidence

GRADE SCALE DECISIONS

Atkins. BMC Health Services Research 2005Atkins. BMC Health Services Research 2005Atkins. BMC Health Services Research 2005Atkins. BMC Health Services Research 2005

4

3

2

1

RecommendationBalance Benefit vs HarmRater

8

7

6

5

12

11

10

9

13

17

16

15

Toss up

Do it

Don't do it

Probably do it

Do it

Do it

Do it

Probably do it

Probably do it

Probably do it

Probably do it

Do it

Toss up

Don't do it

Toss up

Probably do it

Net benefit

Net benefit probably

Uncertain net benefit

Net benefit

Net benefit

Net benefit

Net benefit

Trade offs

Trade offs

Net benefit

Net benefit

Net benefit

Net benefit

Not net benefit

Uncertain net benefit

Net benefit 14

Reasons:• Lack of information in

summaries• Missing baseline risk• What is sparse information?

Reasons:• Lack of information in

summaries• Missing baseline risk• What is sparse information?

INTERPRETATIONSINTERPRETATIONS

I believe magnesium has now been shown to be beneficial for I believe magnesium has now been shown to be beneficial for patients during the post-MI periodpatients during the post-MI period (SD-SA) (SD-SA)

Rev 8Rev 8

Rev 7Rev 7

Rev 6Rev 6

Rev 5Rev 5

Rev 4Rev 4

Rev 3Rev 3

Rev 2Rev 2

Ag

DA

Ag

Ag

DA

DA

SA

Ag

DA

-

Ag

Ag

DA

-

SA

Ag

SD

-

Ag

Ag

DA

-

Ag

AgRev 1Rev 1

II22

Rand. ORRand. OR

Fixed ORFixed OR

Ag

DA

Ag

Ag

SD

SD

Ag

Ag

59%59%

0.75 (0.61-0.92)0.75 (0.61-0.92)

1.01 (0.96-1.07)1.01 (0.96-1.07)

69,50569,505

59%59%

0.65 (0.48-0.87)0.65 (0.48-0.87)

1.02 (0.96-1.08)1.02 (0.96-1.08)

63,04763,047

-

DA

Ag

Ag

Ag

-

SA

Ag

21%21%

0.66 (0.53-0.81)0.66 (0.53-0.81)

0.64 (0.52-0.79)0.64 (0.52-0.79)

3,6853,685

56%56%

0.38 (0.21-0.66)0.38 (0.21-0.66)

0.40 (0.28-0.61)0.40 (0.28-0.61)

597597

0%0%

0.40 (0.18-0.86)0.40 (0.18-0.86)

0.40 (0.19-0.83)0.40 (0.19-0.83)

415415

SD

DA

-

-

SD

SD

Ag

SD

n/an/a

n/an/a

n/an/a

111111NN

1-231-231-201-201-101-101-51-51-31-311# RCTs# RCTs

CONCLUSION• The explosion of meta-analyses has occurred The explosion of meta-analyses has occurred

because of “apparently simple recipes”because of “apparently simple recipes”

• However, a meta-analysis is an observational study However, a meta-analysis is an observational study with potential errors at many levelswith potential errors at many levels

Authors of meta-analyses need to be appropriately Authors of meta-analyses need to be appropriately trainedtrained

Readers need to be Readers need to be appropriatelyappropriately skeptical skeptical

• The benefits of meta-analyses may be lost if current The benefits of meta-analyses may be lost if current growth remains unrestricted and becomes a growth remains unrestricted and becomes a malignancymalignancy

• The explosion of meta-analyses has occurred The explosion of meta-analyses has occurred because of “apparently simple recipes”because of “apparently simple recipes”

• However, a meta-analysis is an observational study However, a meta-analysis is an observational study with potential errors at many levelswith potential errors at many levels

Authors of meta-analyses need to be appropriately Authors of meta-analyses need to be appropriately trainedtrained

Readers need to be Readers need to be appropriatelyappropriately skeptical skeptical

• The benefits of meta-analyses may be lost if current The benefits of meta-analyses may be lost if current growth remains unrestricted and becomes a growth remains unrestricted and becomes a malignancymalignancy

Canadian Academy of Sport MedicineL’Académie Canadienne de Médecine du Sport