Systematic Review, Meta-Analysis, Clinical Practice Guidelines…

Tools, TSH and Tricks of the Trade…

Larry Young

Sheri Keitz

Teaching Objectives

• Understand what systematic reviews can and cannot do for you

• To learn to love forest plots• To review principles of when it is appropriate to

combine studies • Interpret the results of a meta-analysis including

forest plots• Become familiar with the GRADE approach and

its use in creating clinical practice guidelines and implementing evidence

Definitions

• A systematic review is a summary of research that addresses a focused clinical question in a systematic, reproducible manner.

• A meta-analysis is a statistical pooling or aggregation of estimate of effect.

SR MA

Users’ Guide to the Medical Literature. Guyatt et al.

Advantages of Meta-Analyses

• More representative of the total body of evidence

• Save time

• Increase precision of results

• Include a greater range of patients, potentially improving generalizability

• Provide an assessment of confidence in results

The Systematic Review

Hierarchy of Evidence Pyramid

Unsystematic Clinical Experience

Case-Control

Case Series

Observational Studies

(Cohort)

RCT

Systematic Review/Meta-analysis

Ask

Acquire

Appraise

Apply

Assess

Clinical

dilemma

Evidence-based

medicine

cycle

Values &

PreferencesHierarchy of

Evidence

THE BEST WAY TO UNDERSTAND JUST ABOUT ANYTHING IN MEDICINE IS TO REFLECT ON A PATIENT…

The Setting

• You are covering the walk in clinic for one of your colleagues.

• A staff member, who usually sees one of your residents, is here for follow up of her lab results.

Mrs. SR

• 67 year old retired 3rd GRADE teacher

• PMH: hypertension and osteopenia

• Weight gain, dry skin and fatigue for the last 6 months; quality of life has declined – less active, new knee pain

• Normal physical examination

• BP 158/92

• TSH 9.5; normal T3 and T4

• You diagnose subclinical hypothyroidism.

She wonders if thyroid replacement will help with her symptoms

• P : non-pregnant patients with subclinical hypothyroidism

• I : thyroid hormone replacement

• C : no treatment, placebo

• O : quality of life, thyroid related symptoms, HTN, worsening osteopenia

• T : therapy

• T : randomized clinical trial, meta-analysis of RCTs

She wonders if thyroid replacement will help with her symptoms

• P : non-pregnant patients with subclinical hypothyroidism

• I : thyroid hormone replacement

• C : no treatment, placebo

• O : quality of life, thyroid related symptoms, HTN, worsening osteopenia

• T : therapy

• T : randomized clinical trial, meta-analysis of RCTs

Acquire

• 21 studies

• 2192 patients

• RCTs comparing thyroid hormone replacement to placebo or no treatment

Ask

Acquire

Appraise

Apply

Assess

Clinical

dilemma

Evidence-based

medicine

cycle

Values &

PreferencesHierarchy of

Evidence

Appraisal

Let’s appraise the article together… in 5 minutes?

Rea

d A

bst

ract

… 2

m

inu

tes

Assessing the Credibility of the Systematic Review Process

Did the review address a focused clinical question (i.e.

can be framed in PICO format)?

1

Was the search for relevant studies detailed and

exhaustive?

2

Was the risk of bias of the primary studies assessed? 3

Did the review address possible explanations of

between-study differences in results using prespecified

hypotheses?

4

Were selection and assessment of studies reproducible? 5

Did the review describe a process to assess confidence

in effect estimates? (e.g. GRADE tool to assess quality of

the body of evidence)

6

Understanding the Summary Estimate of a Meta-analysis

What is the magnitude of treatment effect? (what is the

pooled estimate?)

7

How precise are the results? (i.e. confidence interval

around the pooled estimate)

8

Rating Confidence in the Estimates (the Quality of a Body of Evidence)How serious is the risk of bias in the body of evidence? 9

Are the results consistent across studies? (i.e.

heterogeneity or inconsistency)

10

Do the results directly apply to my patient? (i.e. PICO,

generalizability, indirectness)

11

Is there a concern about reporting or publication bias? 12

Are there reasons to increase or decrease the confidence

of the rating? (Randomized trials start high and

observational studies start low)

13

Overall, what is the quality of the body of evidence by

outcome? (High, moderate, low, very low)

14

How can I apply the results to my patient care?

Did the review present results that are ready for clinical

application? (e.g. patient important outcomes, absolute

benefit /risk)

15

Are the study patients similar to my patient and are

likely benefits worth potential harms/costs?

16

Ap

pra

isal

… in

yo

ur

bin

der

Assessing the Credibility of the Systematic Review ProcessDid the review address a focused clinical

question (i.e. can be framed in PICO format)?

1

Was the search for relevant studies detailed

and exhaustive?

2

Was the risk of bias of the primary studies

assessed?

3

Did the review address possible explanations

of between-study differences in results using

prespecified hypotheses?

4

Were selection and assessment of studies

reproducible?

5

Did the review describe a process to assess

confidence in effect estimates? (e.g. GRADE

tool to assess quality of the body of

evidence)

6

Appraisal – Did the Meta-Analysts Do a Good Job?

Did the review address a focused clinical question (i.e. can be framed in PICO format)?

Was the search for relevant studies detailed and exhaustive?

Was the risk of bias of the primary studies assessed?

Did the review address possible explanations of between-study differences in results using prespecified hypotheses?

Were selection and assessment of studies reproducible?

Did the review describe a process to assess confidence in effect estimates? (e.g. GRADE tool to assess quality of the body of

evidence)

Assessing the Credibility of the Systematic Review ProcessDid the review address a focused clinical

question (i.e. can be framed in PICO format)?

1

Was the search for relevant studies detailed

and exhaustive?

2

Was the risk of bias of the primary studies

assessed?

3

Did the review address possible explanations

of between-study differences in results using

prespecified hypotheses?

4

Were selection and assessment of studies

reproducible?

5

Did the review describe a process to assess

confidence in effect estimates? (e.g. GRADE

tool to assess quality of the body of

evidence)

6

Appraisal

What are the results

• Everyone: look at Figure 1 (page 1354).

• Let’s look together at the primary outcomes

Page 1354

Page 1354

Should we pool the results?Heterogeneity??? What the fork?

Expert on Heterogeneity

https://www.youtube.com/watch?v=HwqHUSOHPvU

Is pooling sensible?

• A priori determination of whether it makes sense to combine on a principled basis.

• You have not looked at the data yet…

The Common Sense Test

Is pooling sensible?

• MA of all kinds of treatments for all cancers?

• MA of steroid treatments for COPD?

• MA of studies looking at CT-angiogram for diagnosis of pulmonary embolism?

Does combining RESULTS make sense?

• Comes from a review of dots and lines

Favors New Treatment Favors ControlNo Difference

Result 1

Eye-ball test

Favors New Treatment Favors ControlNo Difference

Can we combine?

Favors New Treatment Favors ControlNo Difference

Result 2

Eye-ball test

Favors New Treatment Favors ControlNo Difference

Can we combine?

Favors New Treatment Favors ControlNo Difference

Result 3

Eye-ball test

Favors New Treatment Favors ControlNo Difference

Can we combine?

Okay: What is the “Eye-ball test”?

Eye-ball test

• Visual impression of degree of overlap of each study’s results (find a common neighborhood of truth)

• Visual impression of where the estimates of study “effects” themselves lie (where are the dots)

c

Favors New Treatment Favors ControlNo Difference

Favors New Treatment Favors ControlNo Difference

Back to result 3: if not, why not?

Why might things be different

• P: more or less severe symptoms

• I: types of replacement (and duration)

• C: active control / placebo

• O: different scales for QOL, fatigue, others

• T: treatment question

• T: RCT or other study designs

I2 what?

• From 0 to 100

• The variability that is due to a real difference

• I2 = 0

• I2 = 100

• What is the range of possible values for I2

• Explain in a sentence?

• Is the variability between studies due ONLY to chance?

• Is the variability between studies due ONLY to a real difference in treatment?

Favors New Treatment Favors ControlNo Difference

Result 1: what is I2

Favors New Treatment Favors ControlNo Difference

Result 2: what is I2

Favors New Treatment Favors ControlNo Difference

Result 3: what is I2

Back to Results

BACK TO THE PAPER

BUT HOW MUCH CONFIDENCE CAN WE HAVE IN THIS BODY OF EVIDENCE?

Rating Confidence in the Estimates (the Quality of a Body of Evidence)

How serious is the risk of bias in the body

of evidence?

9

Are the results consistent across studies?

(i.e. heterogeneity or inconsistency)

10

Do the results directly apply to my

patient? (i.e. PICO, generalizability,

indirectness)

11

Is there a concern about reporting or

publication bias?

12

Are there reasons to increase or decrease

the confidence of the rating?

(Randomized trials start high and

observational studies start low)

13

Overall, what is the quality of the body of

evidence by outcome? (High, moderate,

low, very low)

14

How serious is the risk of bias?

Do these look familiar?

Remember, LOW values are GOOD

Rating Confidence in the Estimates (the Quality of a Body of Evidence)

How serious is the risk of bias in the body

of evidence?

9

Are the results consistent across studies?

(i.e. heterogeneity or inconsistency)

10

Do the results directly apply to my

patient? (i.e. PICO, generalizability,

indirectness)

11

Is there a concern about reporting or

publication bias?

12

Are there reasons to increase or decrease

the confidence of the rating?

(Randomized trials start high and

observational studies start low)

13

Overall, what is the quality of the body of

evidence by outcome? (High, moderate,

low, very low)

14

Rating Confidence in the Estimates (the Quality of a Body of Evidence)

How serious is the risk of bias in the body

of evidence?

9

Are the results consistent across studies?

(i.e. heterogeneity or inconsistency)

10

Do the results directly apply to my

patient? (i.e. PICO, generalizability,

indirectness)

11

Is there a concern about reporting or

publication bias?

12

Are there reasons to increase or decrease

the confidence of the rating?

(Randomized trials start high and

observational studies start low)

13

Overall, what is the quality of the body of

evidence by outcome? (High, moderate,

low, very low)

14

Does This Match Our PICOTT?

• P : non-pregnant patients with subclinical hypothyroidism

• I : thyroid hormone replacement

• C : no treatment, placebo

• O : quality of life; thyroid related symptoms, HTN, worsening osteopenia

• T : therapy

• T : randomized clinical trial, meta-analysis of RCTs

Rating Confidence in the Estimates (the Quality of a Body of Evidence)

How serious is the risk of bias in the body

of evidence?

9

Are the results consistent across studies?

(i.e. heterogeneity or inconsistency)

10

Do the results directly apply to my

patient? (i.e. PICO, generalizability,

indirectness)

11

Is there a concern about reporting or

publication bias?

12

Are there reasons to increase or decrease

the confidence of the rating?

(Randomized trials start high and

observational studies start low)

13

Overall, what is the quality of the body of

evidence by outcome? (High, moderate,

low, very low)

14

Is there a concern about reporting or publication bias?

• Positive studies may be as much as 3 times more likely to be published than negative studies.

• Usually exaggerate treatment effect

• Funnel plot

• Look for asymmetry in the included studies

Result

Stu

dy

size

Result

Stu

dy

size

Result

Is there a concern about reporting or publication bias?

Rating Confidence in the Estimates (the Quality of a Body of Evidence)

How serious is the risk of bias in the body

of evidence?

9

Are the results consistent across studies?

(i.e. heterogeneity or inconsistency)

10

Do the results directly apply to my

patient? (i.e. PICO, generalizability,

indirectness)

11

Is there a concern about reporting or

publication bias?

12

Are there reasons to increase or decrease

the confidence of the rating?

(Randomized trials start high and

observational studies start low)

13

Overall, what is the quality of the body of

evidence by outcome? (High, moderate,

low, very low)

14

Are there reasons to increase or decrease the confidence of the rating?

X is GOODSummary of Findings Table

No downgrade; inconsistency due to one trial that showed effect of placebo.

Rating Confidence in the Estimates (the Quality of a Body of Evidence)

How serious is the risk of bias in the body

of evidence?

9

Are the results consistent across studies?

(i.e. heterogeneity or inconsistency)

10

Do the results directly apply to my

patient? (i.e. PICO, generalizability,

indirectness)

11

Is there a concern about reporting or

publication bias?

12

Are there reasons to increase or decrease

the confidence of the rating?

(Randomized trials start high and

observational studies start low)

13

Overall, what is the quality of the body of

evidence by outcome? (High, moderate,

low, very low)

14

For Primary Outcomes: High

Rating Confidence in the Estimates (the Quality of a Body of Evidence)

How serious is the risk of bias in the body

of evidence?

9

Are the results consistent across studies?

(i.e. heterogeneity or inconsistency)

10

Do the results directly apply to my

patient? (i.e. PICO, generalizability,

indirectness)

11

Is there a concern about reporting or

publication bias?

12

Are there reasons to increase or decrease

the confidence of the rating?

(Randomized trials start high and

observational studies start low)

13

Overall, what is the quality of the body of

evidence by outcome? (High, moderate,

low, very low)

14

Rating Confidence in the Estimates (the Quality of a Body of Evidence)

How serious is the risk of bias in the body

of evidence?

9

Are the results consistent across studies?

(i.e. heterogeneity or inconsistency)

10

Do the results directly apply to my

patient? (i.e. PICO, generalizability,

indirectness)

11

Is there a concern about reporting or

publication bias?

12

Are there reasons to increase or decrease

the confidence of the rating?

(Randomized trials start high and

observational studies start low)

13

Overall, what is the quality of the body of

evidence by outcome? (High, moderate,

low, very low)

14

For Primary Outcomes: High

We already did this!

CLINICAL PRACTICE GUIDELINES DEVELOPMENT

Evidence Recommendations

Guideline Development Committee

YOU!

Three Groups

ABlood

pressure

B Fatigue

CBMI

Look at e-Table 4

TABLE VOTE Make recommendation using

handout on your table

YES

NO

WEAK

STRONG

OR OR

Five Ten Minutes

A: Blood Pressure

X is GOODNo difference

Blood PressureMake Recommendation - VOTE

YES

NO

OR

Blood PressureMake Recommendation - VOTE

YES

NO

WEAK

STRONG

OR OR

B: Fatigue

X is GOODDowngrade due to only one trial.

No difference

FatigueMake Recommendation - VOTE

YES

NO

OR

FatigueMake Your Recommendation - VOTE

YES

NO

WEAK

STRONG

OR OR

C: BMI

BMI

X is GOODNo downgrade; inconsistency due to one trial that showed effect of placebo.

No difference

BMIMake Recommendation - VOTE

YES

NO

OR

BMI Make Recommendation- VOTE

YES

NO

WEAK

STRONG

OR OR

Back to Mrs. SR…

• Among nonpregnant adults with subclinical hypothyroidism, the use of thyroid hormone therapy was not associated with improvements in general quality of life or thyroid-related symptoms.

• We advise her that this medication will not improve her symptoms and discuss alternative approaches to address her ongoing concerns.

Anyone Recognize This?

• GRADE: a process for rating the quality of the best available evidence and developing health care recommendations.

• Why did we add this content? – It is a good way to bring many concepts together

and make explicit how evidence can be incorporated on a broader scale

– It is everywhere

GRADE: adopted by over 100 organizations

worldwide

Take Home Messages

• Systematic Reviews and Meta-analyses are needed to inform practice and can be used to drive decisions and actions.

• GRADE is a tool with explicit rules that is transparent and sensible for users of evidence and makers of clinical practice guidelines

• Two components of the GRADE process

– Quality of a body of evidence (certainty, confidence)

– Strength of recommendation: yes/no; weak/ strong

• When determining confidence in results: RCTs start high and Observational studies start low (hierarchy of evidence)

Take Home Messages

• Not all evidence is created equalA Hierarchy of evidence helps us differentiate information more likely to be valid or true

• Evidence alone is never enoughDecisions are made with our patients, guided by values and preferences.

Final Take Home Message:Cookie Monster is awesome