Simple hierarchies are (too) simplistic

Simple hierarchies are (too) simplistic

STUDY DESIGN Randomized Controlled

Trials Cohort Studies and

Case Control Studies Case Reports and Case

Series, Non-systematic observations

BIAS

Expert Opinion

Expert Opinion

Schünemann & Bone, 2003

Likelihood of and confidence in an outcome

Determinants of confidence• RCTs • observational studies • 5 factors that can lower quality

1. limitations in detailed study design and execution (risk of bias criteria)

2. Inconsistency (or heterogeneity)3. Indirectness (PICO and applicability)4. Imprecision5. Publication bias

• 3 factors can increase quality1. large magnitude of effect2. opposing plausible residual bias or confounding3. dose-response gradient

1. Design and Execution/Risk of BiasLimitation in observational studies Explanations

Failure to develop and apply appropriate eligibility criteria (inclusion of control population)

• under- or over-matching in case-control studies

• selection of exposed and unexposed in cohort studies from different populations

Flawed measurement of both exposure and outcome

• differences in measurement of exposure (e.g. recall bias in case- control studies)

• differential surveillance for outcome in exposed and unexposed in cohort studies

Failure to adequately control confounding

• failure of accurate measurement of all known prognostic factors

• failure to match for prognostic factors and/or adjustment in statistical analysis

Incomplete or inadequately short follow-up

1. Design and Execution/Risk of BiasLimitations in RCTs

lack of concealment

intention to treat principle violated

inadequate blinding

loss to follow-up

early stopping for benefit

selective outcome reporting

Design and Execution/RoB

From Cates , CDSR 2008

Design and Execution/RoB

Overall judgment required

YesNoDon’t know or undecided

Who believes the risk of bias is of concern?

Detailed study design and execution

Mortality, cancer andanticoagulation

Akl E, Barba M, Rohilla S, Terrenato I, Sperati F, Schünemann HJ. “Anticoagulation for the long term treatment of venous thromboembolism in patients with cancer”. Cochrane Database Syst Rev. 2008 Apr 16;(2):CD006650.

Five trials

YesNoDon’t know or undecided

Who believes the risk of bias is of concern?

2. Inconsistency of results(Heterogeneity)

• if inconsistency, look for explanation– patients, intervention, comparator, outcome

• if unexplained inconsistency lower quality

Reminders for immunization uptake

Inconsistency

• I2

• P-value• Overlap in CI• Difference in point estimates

3. Directness of Evidencegeneralizability, transferability, applicability

• differences in– populations/patients (HIC – L/MIC, patients with HIV – all

patients)– interventions (new fluroquinolones - old)– comparator appropriate (newer antibx – old)– outcomes (important – surrogate; signs and symptoms –

mortality)• indirect comparisons

– interested in A versus B– have A versus C and B versus C– Cryo + antibiotics versus no intervention versus Cryo versus

no intervention

EVIDENCE PROFILEQuestion: Cyrotherapy with antibiotics vs no antibiotics for histologically confirmed CIN

1 All rates presented at 12 months with assumption that events would occur within this time frame.2 Indirect analysis between single arm observational studies

Quality assessment No of patients Effect

Quality ImportanceNo of studies Design Limitations Inconsistency Indirectness Imprecision Other

Cryotherapy with

antibioticsNo

antibioticsRelative(95% CI) Absolute

Major infection (follow-up 12 months1; requiring hospitalisation or blood transfusion)16 observation

al studiesno serious limitations

no serious inconsistency

serious2 no serious imprecision

none0/1600

(0%)10/4573 (0.22%)

RD 0 (0 to 0)

0 fewer per 1000 OOO IMPORTANT

Resource use - not measured

All severe adverse events (follow-up 12 months; (major infections and bleeding, pelvic inflammatory disease, stenosis, etc )17 observation

al studiesno serious limitations

no serious inconsistency

serious2 no serious imprecision

none0/1705

(0%)22/5142 (0.43%)

RD 0 (0 to 0)

0 fewer per 1000 OOO IMPORTANT

4. Publication Bias

• Should always be suspected– Only small “positive” studies– For profit interest– Various methods to evaluate – none perfect, but

clearly a problem

Egger M, Smith DS. BMJ 1995;310:752-54 19

I.V. Mg in acute

myocardial infarction

Publication bias

Meta-analysisYusuf S.Circulation 1993

ISIS-4Lancet 1995

Egger M, Cochrane Colloquium Lyon 2001 20

Funnel plotS

tand

ard

Err

or

Odds ratio0.1 0.3 1 3

3

2

1

0

100.6

Symmetrical:No publication bias

Egger M, Cochrane Colloquium Lyon 2001 21

Funnel plotS

tand

ard

Err

or

Odds ratio0.1 0.3 1 3

3

2

1

0

100.6

Asymmetrical:Publication bias?

0.4

5. Imprecision• Small sample size

– small number of events

• Wide confidence intervals– uncertainty about magnitude of effect

Example: Immunization in children

For systematic reviews

• If the 95% CI excludes a relative risk (RR) of 1.0 and the total number of events or patients exceeds the OIS criterion, precision is adequate. If the 95% CI includes appreciable benefit or harm (we suggest a RR of under 0.75 or over 1.25 as a rough guide) rating down for imprecision may be appropriate even if OIS criteria are met.

Optimal information size

• We suggest the following: if the total number of patients included in a systematic review is less than the number of patients generated by a conventional sample size calculation for a single adequately powered trial, consider rating down for imprecision. Authors have referred to this threshold as the “optimal information size” (OIS)

What can raise quality?

1. large magnitude can upgrade (RRR 50%/RR 2)– very large two levels (RRR 80%/RR 5)– criteria

• everyone used to do badly• almost everyone does well

– parachutes to prevent death when jumping from airplanes

Reminders for immunization uptake

What can raise quality?2. dose response relation

– (higher INR – increased bleeding)– childhood lymphoblastic leukemia

• risk for CNS malignancies 15 years after cranial irradiation• no radiation: 1% (95% CI 0% to 2.1%) • 12 Gy: 1.6% (95% CI 0% to 3.4%) • 18 Gy: 3.3% (95% CI 0.9% to 5.6%)

3. all plausible residual confounding may be working to reduce the demonstrated effect or increase the effect if no effect was observed

All plausible residual confoundingwould result in an overestimate of effect

Hypoglycaemic drug phenformin causes lactic acidosis

The related agent metformin is under suspicion for the same toxicity.

Large observational studies have failed to demonstrate an association– Clinicians would be more alert to lactic acidosis in

the presence of the agent• Vaccine – adverse effects

Quality assessment criteria Study design

Initial quality of a body of evidence

Lower if Higher if Quality of a body of evidence

Randomised trials

High Risk of Bias

Inconsistency

Indirectness

Imprecision

Publication bias

Large effect Dose response All plausible residual confounding & bias -Would reduce a demonstrated effect -Would suggest a spurious effect if no effect was observed

A/High (four plus: )

B/Moderate (three plus: )

Observational studies

Low C/Low (two plus: )

D/Very low (one plus: )

Overall quality of a body of evidence

• The quality of evidence reflects the extent of our confidence that the estimates of an effect are adequate to support a particular decision or recommendation.

• Guideline developers must specify and determine importance of all relevant outcomes

• Overall quality of evidence is based on the lowest quality of all critical outcomes

Meta-analyses of several critical and important outcomes (one PICO)

Better Relative RiskWorse

0.5 0.75 1 1.25 1.5

SAE (critical)

Nausea (important)

Myo. Infarct. (critical)

Mortality (critical) High

Low Due to imprecision and risk of bias

Moderate Due to imprecision

High

Overall Quality of Evidence: Moderate based on critical outcomes

Low

Meta-analyses of several critical outcomes (one PICO)


0.5 0.75 1 1.25 1.5

SAE

Stroke

Dis. Specific QoL

Mortality

Threshold of acceptable harm for strong

recommendation based on sure benefit in mortality and stroke

High

High

Moderate Due to imprecision

High

Overall Quality of Evidence: High

Meta-analyses of several critical outcomes (one PICO)


0.5 0.75 1 1.25 1.5

SAE

Stroke

Dis. Specific QoL

Mortality

Overall Quality of Evidence:

High

Moderate due to risk of bias

High

High

Moderate

Threshold of acceptable harm for strong

recommendation based on sure benefit in mortality and stroke

Interpretation of grades of evidence

• /A/High: Further research is very unlikely to change confidence in the estimate of effect.

• /B/Moderate: Further research is likely to have an important impact on confidence in the estimate of effect and may change the estimate.

• /C/Low: Further research is very likely to have an important impact on confidence in the estimate of effect and is likely to change the estimate.

• /D/Very low: We have very little confidence in the effect estimate: Any estimate of effect is very uncertain.

Agenda8.30-8.45 Introduction to the course8.45-9.15 Overview of guideline development9.15-10.15 The GRADE approach: introduction10.15-10.30 Break10.30-11.30 Asking a key question, specific outcomes – small group work11.30-12.30 The GRADE approach: assessing the quality of evidence12.30-13.15 Lunch13.15-14.45 Grading quality of evidence – small group work14.45-15.15 Moving from evidence to recommendations: theoretical considerations15.15-15.30 Break15.30-16 Making recommendations – small group work16-16.30 Report from small groups and practical considerations of group process

MOVING FROM EVIDENCE TO RECOMMENDATIONS

Holger (contributions from Yngve Falck-Ytter)

Strength of recommendation

“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.” • Strong or conditional

Evidence based healthcare decisions

Research evidence

Population/societalvalues

and preferences

(Clinical) state and circumstances

Expertise

Haynes et al. 2002


Research evidence

Population/societalvalues

and preferences

Expertise

Evidence of baseline risk and context for a given population

Schunemann et al. in prep


Research evidence

Expertise


Evidence of population & societal values

& preferences

Schunemann et al. in prep


Expertise


Evidence of population & societal values

& preferences

Evidence of effectsSchunemann et al. in prep

From evidence to recommendations

RCTObser-vational study

High level recommen-

dation

Lower level recommen-

dation

Old system

Quality of evidence

Balance between benefits, harms & burdens

Patients’ values &

preferences

GRADE

Resource use

Determinants of the strength of recommendation

Factors that can strengthen a recommendation

Comment

Quality of the evidence The higher the quality of evidence, the more likely is a strong recommendation.

Balance between desirable and undesirable effects

The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely weak recommendation warranted.

Values and preferences The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.

Costs (resource allocation) The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted

Balancing desirable and undesirable consequences

Conditional

Strong For Against

Effects & $ x

values

Effects & $ x

values

Holger Schunemann

this is a new slide


↑ Allergic reactions

↑ Local skin reactions

↑ Nausea↑ Resources

↑ QoL ↓ Death

↓ Morbidity

↑ herd immunity

Conditional

Strong For Against

Effect x value Effect x value

Holger Schunemann

This is a new slide

↑ Allergic

reactions

↑ Local skin

reactions

↑ Nausea

↑ Resources

↑ QoL↓ Death

↓ Morbidity↑ herd

immunityConditional

Strong For Against


Holger Schunemann

this slide has changed

↑ Allergic reactions↑ Local skin reactions

↑ Nausea

↑ Resources↑ QoL ↓ Death

↓ Morbidity

↑ herd immunity

Conditional

Strong For Against


Holger Schunemann



↑ Allergic

reactions

↑ Local skin

reactions

↑ Nausea

↑ Resources

↑ QoL

↓ Death

↓

Morbidity

↑ herd

immunity

Conditional

Strong For Against

Holger Schunemann



↑ Allergic reactions ↑ Local skin

reactions

↑ Nausea

↑ Resources

↑ QoL

↓ Death

↓ Morbidity

↑ herd immunity

Conditional

Strong For Against

Holger Schunemann

This is a new slide

Implications of a strong recommendation

• Policy makers: The recommendation can be adapted as a policy in most situations

• Patients: Most people in this situation would want the recommended course of action and only a small proportion would not

• Clinicians: Most patients should receive the recommended course of action

Implications of a conditional recommendation• Policy makers: There is a need for

substantial debate and involvement of stakeholders

• Patients: The majority of people in this situation would want the recommended course of action, but many would not

• Clinicians: Be more prepared to help patients to make a decision that is consistent with their own values/decision aids and shared decision making

Case scenario

A 13 year old girl who lives in rural Indonesia presented with flu symptoms and developed severe respiratory distress over the course of the last 2 days. She required intubation. The history reveals that she shares her living quarters with her parents and her three siblings. At night the family’s chicken stock shares this room too and several chicken had died unexpectedly a few days before the girl fell sick.

Methods – WHO Rapid Advice Guidelines for Avian Flu

Applied findings of a recent systematic evaluation of guideline development for WHO/ACHR

Group composition (including panel of 13 voting members): clinicians who treated influenza A(H5N1) patients infectious disease experts basic scientists public health officers methodologists

Independent scientific reviewers: Identified systematic reviews, recent RCTs, case series, animal

studies related to H5N1 infection

Oseltamivir for Avian FluSummary of findings: • No clinical trial of oseltamivir for treatment of H5N1

patients.• 4 systematic reviews and health technology

assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza. – Hospitalization: OR 0.22 (0.02 – 2.16)– Pneumonia: OR 0.15 (0.03 - 0.69)

• 3 published case series. • Many in vitro and animal studies. • No alternative that was more promising at present.• Cost: 40$ per treatment course

Evidence profile

No of studies(Ref) Design Limitations Consistency Directness Other

considerations Oseltamivir Placebo Relative(95% CI )

Absolute(95% CI )

Mortality 0 - - - - - - - - - 9

5(TJ 06)

Randomised trial

No limitations One trial only Major uncertainty (-2)1

Imprecise or sparse data (-1)

- - OR 0.22(0.02 to 2.16)

- Very low

6

0 - - - - - - - - - - 7

5(TJ 06)

Randomised trial

No limitations One trial only Major uncertainty (-2)1

Imprecise or sparse data (-1)2

2/982(0.2%)

9/662(1.4%)

RR 0.149(0.03 to 0.69)

- Very low

8

53

(TJ 06)(DT 03)

Randomised trials

No limitations4 Important inconsistency(-1)5

Major uncertainty (-2)1

- - - HR 1.303

(1.13 to 1.50)-

Very low5

26

(TJ 06)Randomised trials

No limitations -7 Major uncertainty (-2)1

None - - - WMD -0.738

(-0.99 to -0.47)

Low4

0 - - - - - - - - - - 4

0 - - - - - - - - - - 7

09 - - - - - - - - - - 7

311

(TJ 06)Randomised trials

No limitations -12 Some uncertainty (-1)13

Imprecise or sparse data (-1)14

- - OR range15

(0.56 to 1.80)-

Low

0 - - - - - - - - - - 4

I mportance

Summary of findings

Cost of drugs

Outbreak control

Resistance

Serious adverse effects (Mention of significant or serious adverse effects)

Minor adverse effects 10 (number and seriousness of adverse effects)

Viral shedding (Mean nasal titre of excreted virus at 24h)

Duration of disease (Time to alleviation of symptoms/median time to resolution of symptoms – influenza cases only)

Duration of hospitalization

LRTI (Pneumonia - influenza cases only)

Healthy adults:

Hospitalisation (Hospitalisations from influenza – influenza cases only)

Quality assessmentNo of patients Effect

Quality

Oseltamivir - H5N1

-

-

Quality across

endpoints

PICO & Confidence in estimates

Benefits vs harms

Values and preferences

Resource utilization

From evidence to recommendation

Factors that can strengthen a recommendation

Comment

Quality of the evidence Very low quality evidence

Balance between desirable and undesirable effects

Uncertain, but small reduction in relative risk still leads to large absolute effect

Values and preferences Little variability and clear

Costs (resource allocation) Low cost under non-pandemic conditions

Example: Oseltamivir for Avian FluRecommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence).

Remarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment.

Schunemann et al. The Lancet ID, 2007

?

Other explanationsRemarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time.

The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.

GRADE Grid

Values and preferences

• Implicit value judgments in recommendations

• Trade-offs: example prevention of VTE in surgery– Thrombotic events

• Deep vein thrombosis, pulmonary embolism– Bleeding events

• Gastrointestinal bleeds, operative site bleeds– Inconvenience of injections

• Variability in values and preferences

Case• 77 y/o patient with atrial fibrillation, mild

CHF, HTN, DM and history of stroke (fully recovered)

• Meds: warfarin, antihypertensives, statin, glyburide

• Admitted with nausea/vomiting, then hematemesis; INR 2.5; 1 U blood transfused; EGD: no active bleed, possible Mallory Weiss

• This is his second major bleed since he started warfarin one year ago

CHADS2 score

Acceptable additional bleeds?• Study: Patients at high risk for atrial fibrillation and high

risk of stroke (h/o CHF/MI); internists and cardiologists

• Warfarin decreases risk at cost of increased GI bleeds• Without treatment 100 patients will suffer:

– 12 strokes (six major, six minor), 3 serious GI bleeds in 2 years

• Warfarin would decrease strokes in 100 patients to 4 per 2 years (8 fewer strokes, 4 major, minor)

• How many additional bleeds would you accept in 100 patients over a year, and still be willing to administer/take warfarin?

Strength of recommendation“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.”

Understanding values & preferences necessary to trade-off benefits and downsides

Values and preferences should ideally be informed by systematic reviews, but evidence is often sparse

Example recommendation

ACCP AT9 recommendation:In patients undergoing major orthopedic surgery (e.g., total hip replacement), we suggest the use of LMWH in preference to the other agents.Patients who place a high value on avoiding bleeding complications and a low value on its inconvenience are likely to choose a compression device (IPCD) over the drug options.

Considering resource use

• Allocating resources such as cost• In GRADE: cost = outcome. Challenges include:

– Who pays? Individual vs. society?– Varies widely (even within 1 kilometer) – Opportunity cost differ– Often highly political (e.g., USA)

• Resource use considered last as an additional step

Example 1: ipilimumabfor metastatic melanoma

Estimated absolute effects at 2 years

Outcome Relative effect

(95% CI)

control(per 1000)

ipilimumab(per 1000)

Difference(per 1000)

Certainty of the effect

Death HR 0.68(0.55 to

0.85)850 725 125 fewer deaths

(49 to 202 fewer)

High

Serious immune-

related AEs

RR 7.37(4.42 to

12.3)39 289 249 more SAEs

(134 to 441 more)

High

Quality of life

Acquisition cost

Ipilimumab: $120,000 (12 weeks induction (4 injections))Dacarbazine: $400

Simplified cost effectiveness projection

$ or $ per LY

Dacarbazine induction $400

Ipilimumab induction $120,000

Increase in median survival 3.6 mo (0.3 years)

Incremental cost-effectiveness ($/life year)

~$400,000

Not only cost but also resource use

• Emergency room visits, hospitalizations• Severe immune-mediated events (grade 3 and

4): 86/627 patients with hepatic toxicity, 18/627 with enterocolitis; 5 immune-mediated deaths associated with ipilimumab

• Use of other interventions downstream• Include complete resource use in evidence

profile if possible (e.g., hospital days, nurses hours, drug units dispensed, etc.)

Define perspective taken

• Example 1: single regional health system– Health plan / insurance– Country– International: e.g., WHO

• Perspective should be sufficiently broad– Should include downstream effects– Comprehensive view desirable: e.g., societal

• Includes all cost regardless who bears them

Quality of evidence assessment

• Rarely assessed in the context of RCTs used to inform estimate of effect

• Often derived from other sources of evidence

• More confident in the acquisition cost than the estimate of resources used for serious AEs

Modeling resource use• Modeling results in

cost per unit benefits achieved can be helpful– …but sometimes be misleading and prone to

bias– …relies heavily on assumptions – …debate whether indirect costs, such as lost

productivity should be included• Avoid including cost-utility models directly

in the GRADE evidence profile

SummaryFrom evidence to making recommendations

1. Beyond net benefit, GRADE makes transparent the process on how values & preferences and resource use informs decision making

2. Understanding values & preferences is necessary to make trade-offs between benefits and downsides

3. If necessary, resource use considerations can inform direction and strength of recommendations

Conclusions Guidelines should be based on the best available evidence to

be evidence based GRADE is the approach used by over 65 organizations and

gaining acceptance internationally combines what is known in health research methodology and

provides a structured approach to improve communication Does not avoid judgments but provides framework Criteria for evidence assessment across questions and

outcomes Criteria for moving from evidence to recommendations Transparent, systematic

four categories of quality of evidence two grades for strength of recommendations

Transparency in decision making and judgments is key

Simple hierarchies are (too) simplistic

Documents

Transcript of Simple hierarchies are (too) simplistic