GRADE From Evidence to Recommendations

GRADE FROM EVIDENCE TO RECOMMENDATIONS

Holger Schünemann, MD, PhDProfessorUtrecht, NLSeptember 18 - 19, 2008

1

Evidence based clinical decisions

Research evidence

Patient valuesand preferences

Clinical state and circumstances

Expertise

Equal for allHaynes et al. 2002

The GRADE approach

Separation of 2 issues:1) 4 categories of quality of evidence:

very low, low, moderate, or high quality? methodological quality of evidence likelihood of bias by outcome and across outcomes

2) Recommendation: 2 grades - weak or strong (for or against)? Quality of evidence only one factor

www.GradeWorkingGroup.org

Grades of recommendation:Strength of recommendationsStrong recommendations high quality methods with large precise effect benefits much greater than downsides, or downsides

much greater than benefits we recommend Grade 1

Weak/conditional recommendations lower quality methods benefits not clearly greater or smaller than downsides values and preferences uncertain or very variable we suggest Grade 2

Case scenario

A 13 year old girl who lives in rural Indonesia presented with flu symptoms and developed severe respiratory distress over the course of the last 2 days. She required intubation. The history reveals that she shares her living quarters with her parents and her three siblings. At night the family’s chicken stock shares this room too and several chicken had died unexpectedly a few days before the girl fell sick.

Relevant clinical question?Example from a not so common disease

Clinical question:Population: Avian Flu/influenza A (H5N1) patientsIntervention: Oseltamivir (or Zanamivir) Comparison: No pharmacological interventionOutcomes: Mortality, hospitalizations,

resource use, adverse outcomes, antimicrobial resistanceSchunemann et al. The Lancet ID, 2007

Methods – WHO Rapid Advice Guidelines for management of Avian Flu Applied findings of a recent systematic evaluation

of guideline development for WHO/ACHR

Group composition (including panel of 13 voting members):

clinicians who treated influenza A(H5N1) patients infectious disease experts basic scientists public health officers methodologists

Independent scientific reviewers: Identified systematic reviews, recent RCTs, case series,

animal studies related to H5N1 infection

Evidence Profile

No of studies(Ref) Design Limitations Consistency Directness Other

considerations Oseltamivir Placebo Relative(95% CI )

Absolute(95% CI )

Mortality 0 - - - - - - - - - 9

5(TJ 06)

Randomised trial

No limitations One trial only Major uncertainty (-2)1

Imprecise or sparse data (-1)

- - OR 0.22(0.02 to 2.16)

- Very low

6

0 - - - - - - - - - - 7

5(TJ 06)

Randomised trial


Imprecise or sparse data (-1)2

2/982(0.2%)

9/662(1.4%)

RR 0.149(0.03 to 0.69)

- Very low

8

53

(TJ 06)(DT 03)

Randomised trials

No limitations4 Important inconsistency(-1)5

Major uncertainty (-2)1

- - - HR 1.303

(1.13 to 1.50)-

Very low5

26

(TJ 06)Randomised trials

No limitations -7 Major uncertainty (-2)1

None - - - WMD -0.738

(-0.99 to -0.47)

Low4

0 - - - - - - - - - - 4

0 - - - - - - - - - - 7

09 - - - - - - - - - - 7

311


No limitations -12 Some uncertainty (-1)13


- - OR range15

(0.56 to 1.80)-

Low

0 - - - - - - - - - - 4

I mportance

Summary of findings

Cost of drugs

Outbreak control

Resistance

Serious adverse effects (Mention of significant or serious adverse effects)

Minor adverse effects 10 (number and seriousness of adverse effects)

Viral shedding (Mean nasal titre of excreted virus at 24h)

Duration of disease (Time to alleviation of symptoms/median time to resolution of symptoms – influenza cases only)

Duration of hospitalization

LRTI (Pneumonia - influenza cases only)

Healthy adults:

Hospitalisation (Hospitalisations from influenza – influenza cases only)

Quality assessmentNo of patients Effect

Quality

Oseltamivir for treatment of H5N1 infection:

-

-

Oseltamivir for Girl with Avian FluSummary of findings: No clinical trial of oseltamivir for treatment of

H5N1 patients. 4 systematic reviews and health technology

assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza. Hospitalization: OR 0.22 (0.02 – 2.16) Pneumonia: OR 0.15 (0.03 - 0.69)

3 published case series. Many in vitro and animal studies. No alternative that is more promising at

present. Cost: ~ Euro 40$ per treatment course

Determinants of the strength of recommendation

Factors that can strengthen a recommendation

Comment

Quality of the evidence The higher the quality of evidence, the more likely is a strong recommendation.

Balance between desirable and undesirable effects

The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely weak recommendation warranted.

Values and preferences The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.

Costs (resource allocation) The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted

Example: Oseltamivir for Avian Flu

Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (????? recommendation, very low quality evidence).

Schunemann et al. The Lancet ID, 2007


Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence). Values and PreferencesRemarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment.


Other explanations

Remarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time.

The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.

ACCP: Acute coronary syndromeWould a recommendation ignoring V & P be possible?

For all patients presenting with NSTE ACS, without a clear allergy to aspirin, we recommend immediate aspirin, 75 to 325 mg po, and then daily, 75 to 162 mg po (strong recommendation, high quality evidence).

Value sensitive recommendation

Idiopathic deep venous thrombosis (DVT) is potentially life threatening condition

Patients usually receive blood thinners for one year Continuing therapy will reduce a patients absolute

risk for recurrent DVT by 7% per year for several years

The burdens include: taking a blood thinner daily keeping dietary intake of vitamin K constant blood tests to monitor the intensity of anticoagulation living with increased risk of minor and major bleeding

Value sensitive recommendations→ Patients who are very averse to a recurrent DVT would consider the benefits of avoiding DVT worth the downsides of taking warfarin. Other patients are likely to consider the benefit not worth the harms and burden. For patients with idiopathic DVT, without elevated bleeding risk, we suggest long term warfarin therapy (weak recommendation, high quality evidence).Values and preferences: this recommendation ascribes a low value to bleeding complications and burden from therapy and a high value to avoiding DVTs

Quality assessment criteria Quality of evidence

Study design Lower if Higher if

High Randomised trial Study quality: Serious limitations Very serious limitations I mportant inconsistency Directness: Some uncertainty Major uncertainty Sparse or imprecise data High probability of reporting bias

Strong association: Strong, no plausible confounders Very strong, no major threats to validity Evidence of a Dose response gradient All plausible confounders would have reduced the eff ect

Moderate

Low Observational study

Very low Any other evidence

Strength of recommendation

“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.”

Strong or weak

Quality of evidence & strength of recommendation Linked but no automatism Other factors beyond the quality of

evidence influence our confidence that adherence to a recommendation causes more benefit than harm

Systems/approaches failed to make this explicit

GRADE separates quality of evidence from strength of recommendation

Implications of a strong recommendation Patients: Most people in this situation

would want the recommended course of action and only a small proportion would not

Clinicians: Most patients should receive the recommended course of action

Policy makers: The recommendation can be adapted as a policy in most situations

Implications of a weak recommendation Patients: The majority of people in this

situation would want the recommended course of action, but many would not

Clinicians: Be prepared to help patients to make a decision that is consistent with their own values/decision aids and shared decision making

Policy makers: There is a need for substantial debate and involvement of stakeholders

Evaluating desirable and undesirable effects

Desirable << Undesirable Effects Desirable ?< Undesirable Effects Desirable ?> Undesirable Effects Desirable >> Undesirable Effects

Formulating a recommendation

Against For

Strong

1

Weak ? 2

Weak ? 2

Strong

1

Respiratory disease guidelines ?

Factors determining strength of recommendation


Comment



The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower the certainty for that benefit, the more likely is a weak recommendation.



Values & Preferences

Patients’ perspectives, beliefs, expectations, and goals for health and life.

Underlying processes used in considering the benefits, harms, costs, and inconveniences patients will experience with each management option and the resulting preferences for each option.

Guideline panels should be explicit about the relative value they place on the range of relevant patient-important outcomes. If values and preferences vary widely, a strong recommendation becomes less likely

Example: Patients vary widely in their view of how aversive they find the risk of a stroke versus the risk of a gastrointestinal bleed when deciding about oral anticoagulation for atrial fibrillation.

Relative importance of outcomes and management approaches

Desirable and undesirable effects desirable effects

Mortality improvement in quality of life, fewer

hospitalizations reduction in the burden of treatment reduced resource expenditure

undesirable consequences deleterious impact on morbidity, mortality

or quality of life, increased resource expenditure

Conclusion

clinicians, policy makers need summaries quality of evidence strength of recommendations

explicit rules transparent, informative

GRADE four categories of quality of evidence two grades for strength of recommendations transparent, systematic by and across outcomes applicable to diagnosis wide adoption

Design and Execution

limitations lack of concealment intention to treat principle violated inadequate blinding loss to follow-up early stopping for benefit

Example: RCT suggests that danaparoid sodium is of benefit in treating HIT complicated by thrombosis Key outcome: clinicians’ assessment of when the

thromboembolism had resolved Not blinded – subjective judgement

Consistency of results

consistency of results if inconsistency, look for explanation

patients, intervention, outcome, methods unexplained inconsistency downgrade

quality Bleeding in thrombosis-prophylaxed

hospitalized patients seven RCTs 4 lower, 3 higher risk

Example: Bleeding in the hospital

Dentali et al. Ann Int Med, 2007

Directness of Evidence

indirect comparisons interested in A versus B have A versus C and B versus C formoterol versus salmeterol versus tiotropium

differences in patients (mild versus severe COPD) interventions (all inhaled steroids) outcomes (long-term health-related quality of

life, short –term functional capacity, laboratory exercise, spirometry)

Reporting Bias & Imprecision reporting bias

reporting of studies publication bias

number of small studies reporting of outcomes

small sample size small number of events wide confidence intervals uncertainty about magnitude of effect

Example: Bleeding in the hospital

Dentali et al. Ann Int Med, 2007

What can raise quality?

large magnitude can upgrade (RRR 50%) very large two levels (RRR 80%) common criteria

everyone used to do badly almost everyone does well

Oral anticoagulation for mechanical heart valves

dose response relation (higher INR – increased bleeding)

Questions for you

Are systematic reviews for every recommendation in your guidelines a reality/possibility?

What about cost – how do you deal with cost and how should we deal with it?

Content

Study design – bias Levels/quality of evidence - GRADE

Guidelines/Recommendations

Confidence in evidence

There always is evidence “When there is a question there is

evidence” Better research greater confidence

in the evidence and decisions Evidence alone is never sufficient to

make a clinical decision

Evidence based clinical decisions

Research evidence

Patient valuesand preferences

Clinical state and circumstances

Expertise

Equal for allHaynes et al. 2002

About GRADE

Since 2000 Researchers/guideline developers with

interest in methodology Aim: to develop a common,

transparent and sensible system for grading the quality of evidence and the strength of recommendations

Evaluation of existing systems

GRADE Evidence Profiles

The GRADE approach

Clear separation of 2 issues:1) 4 categories of quality of evidence:



*www.GradeWorking-Group.org

Determinants of quality

RCTs start high observational studies start low what can lower quality?

1. detailed design and execution2. inconsistency3. indirectness4. reporting bias5. imprecision

The GRADE approach

Separation of 2 issues:1) 4 categories of quality of evidence:



www.GradeWorkingGroup.org

Grades of recommendation:Strength of recommendationsStrong recommendations high quality methods with large precise effect benefits much greater than downsides, or downsides

much greater than benefits we recommend Grade 1

Weak/conditional recommendations lower quality methods benefits not clearly greater or smaller than downsides values and preferences uncertain or very variable we suggest Grade 2

Case scenario

A 13 year old girl who lives in rural Indonesia presented with flu symptoms and developed severe respiratory distress over the course of the last 2 days. She required intubation. The history reveals that she shares her living quarters with her parents and her three siblings. At night the family’s chicken stock shares this room too and several chicken had died unexpectedly a few days before the girl fell sick.

Relevant clinical question?Example from a not so common disease

Clinical question:Population: Avian Flu/influenza A (H5N1) patientsIntervention: Oseltamivir (or Zanamivir) Comparison: No pharmacological interventionOutcomes: Mortality, hospitalizations,

resource use, adverse outcomes, antimicrobial resistanceSchunemann et al. The Lancet ID, 2007

Methods – WHO Rapid Advice Guidelines for management of Avian Flu Applied findings of a recent systematic evaluation

of guideline development for WHO/ACHR

Group composition (including panel of 13 voting members):

clinicians who treated influenza A(H5N1) patients infectious disease experts basic scientists public health officers methodologists

Independent scientific reviewers: Identified systematic reviews, recent RCTs, case series,

animal studies related to H5N1 infection

Evidence Profile

No of studies(Ref) Design Limitations Consistency Directness Other

considerations Oseltamivir Placebo Relative(95% CI )

Absolute(95% CI )

Mortality 0 - - - - - - - - - 9

5(TJ 06)

Randomised trial


Imprecise or sparse data (-1)

- - OR 0.22(0.02 to 2.16)

- Very low

6

0 - - - - - - - - - - 7

5(TJ 06)

Randomised trial



2/982(0.2%)

9/662(1.4%)

RR 0.149(0.03 to 0.69)

- Very low

8

53

(TJ 06)(DT 03)

Randomised trials

No limitations4 Important inconsistency(-1)5

Major uncertainty (-2)1

- - - HR 1.303

(1.13 to 1.50)-

Very low5

26


No limitations -7 Major uncertainty (-2)1

None - - - WMD -0.738

(-0.99 to -0.47)

Low4

0 - - - - - - - - - - 4

0 - - - - - - - - - - 7

09 - - - - - - - - - - 7

311


No limitations -12 Some uncertainty (-1)13


- - OR range15

(0.56 to 1.80)-

Low

0 - - - - - - - - - - 4

I mportance

Summary of findings

Cost of drugs

Outbreak control

Resistance

Serious adverse effects (Mention of significant or serious adverse effects)

Minor adverse effects 10 (number and seriousness of adverse effects)

Viral shedding (Mean nasal titre of excreted virus at 24h)

Duration of disease (Time to alleviation of symptoms/median time to resolution of symptoms – influenza cases only)

Duration of hospitalization

LRTI (Pneumonia - influenza cases only)

Healthy adults:

Hospitalisation (Hospitalisations from influenza – influenza cases only)

Quality assessmentNo of patients Effect

Quality

Oseltamivir for treatment of H5N1 infection:

-

-

Oseltamivir for Girl with Avian FluSummary of findings: No clinical trial of oseltamivir for treatment of

H5N1 patients. 4 systematic reviews and health technology

assessments (HTA) reporting on 5 studies of oseltamivir in seasonal influenza. Hospitalization: OR 0.22 (0.02 – 2.16) Pneumonia: OR 0.15 (0.03 - 0.69)

3 published case series. Many in vitro and animal studies. No alternative that is more promising at

present. Cost: ~ Euro 40$ per treatment course

Determinants of the strength of recommendation


Comment



The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely weak recommendation warranted.




Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (????? recommendation, very low quality evidence).



Recommendation: In patients with confirmed or strongly suspected infection with avian influenza A (H5N1) virus, clinicians should administer oseltamivir treatment as soon as possible (strong recommendation, very low quality evidence). Values and PreferencesRemarks: This recommendation places a high value on the prevention of death in an illness with a high case fatality. It places relatively low values on adverse reactions, the development of resistance and costs of treatment.


Other explanations

Remarks: Despite the lack of controlled treatment data for H5N1, this is a strong recommendation, in part, because there is a lack of known effective alternative pharmacological interventions at this time.

The panel voted on whether this recommendation should be strong or weak and there was one abstention and one dissenting vote.

ACCP: Acute coronary syndromeWould a recommendation ignoring V & P be possible?

For all patients presenting with NSTE ACS, without a clear allergy to aspirin, we recommend immediate aspirin, 75 to 325 mg po, and then daily, 75 to 162 mg po (strong recommendation, high quality evidence).

Value sensitive recommendation

Idiopathic deep venous thrombosis (DVT) is potentially life threatening condition

Patients usually receive blood thinners for one year Continuing therapy will reduce a patients absolute

risk for recurrent DVT by 7% per year for several years

The burdens include: taking a blood thinner daily keeping dietary intake of vitamin K constant blood tests to monitor the intensity of anticoagulation living with increased risk of minor and major bleeding

Value sensitive recommendations→ Patients who are very averse to a recurrent DVT would consider the benefits of avoiding DVT worth the downsides of taking warfarin. Other patients are likely to consider the benefit not worth the harms and burden. For patients with idiopathic DVT, without elevated bleeding risk, we suggest long term warfarin therapy (weak recommendation, high quality evidence).Values and preferences: this recommendation ascribes a low value to bleeding complications and burden from therapy and a high value to avoiding DVTs

Background to GRADEpro

People faced with a health care decision need summaries, including information about the quality of evidence Detailed summaries: providing guidance for large

audiences, e.g. guideline panels – detailed profiles

Usable shorter summaries: users of systematic reviews, e.g. clinicians

How do I create a summary of findings table or evidence profile GRADEpro – software to create SoF Import data from RevMan 5 into

GRADEpro Create table – author makes

suggestions about information to present and GRADEs the evidence

Export table from GRADEpro and import into RevMan 5

Creating a new GRADEpro file

Profile groups

Profiles

Profiles: Questions

Importing a RevMan 5 file of a systematic review

Imported data from RevMan 5 file: • outcomes• meta-analyses results• bibliographic information

Managing outcomes to include a maximum of 7

Entering/editing information for dichotomous outcomes

Entering/editing information to grade the quality of the evidence

2. Consistency of results Look for explanation for inconsistency

patients, intervention, comparator, outcome, methods

Judgment variation in size of effect overlap in confidence intervals statistical significance of heterogeneity I2

3. Directness of Evidence

indirect comparisons interested in A versus B have A versus C and B versus C

differences in patients interventions outcomes

Directness of EvidenceTable 5. Sources of likely indirectness of evidenceSource of indirectness Question of interest ExampleIndirect comparison Early emergency department

systemic corticosteroids to treat acute exacerbations in adult patients with asthma

Both oral and intravenous routes are effective but there is no direct comparison of these two routes of administration in adults.

Differences in populations

Anti-leukotrienes plus inhaled glucocorticosteroids vs. inhaled glucocorticosteroids alone to prevent asthma exacerbations and nighttime symptoms in patients with chronic asthma and allergic rhinitis.

Trials that measured asthma exacerbations and nighttime symptoms did not include patients with allergic rhinitis.

Differences in intervention

Avoidance of pet allergens in non-allergic infants or preschool children to prevent development of allergy.

Available studies used multifaceted interventions directed at multiple potential risk factors in addition to pet avoidance.

Differences in outcomes of interest

Intranasal glucocorticosteroids vs. oral H1-antihistamines in children with seasonal allergic rhinitis.

In the available study parents were rating the symptoms and quality of life of their teenage children, instead the children themselves.

4. Reporting Bias

reporting bias reporting of studies

publication bias number of small studies

reporting of outcomesExample: a systematic review of topical treatments for seasonal allergic conjunctivitis showed that patients using topical sodium cromoglycate were more likely to perceive benefit than those using placebo. However, only small trials reported clinically and statistically significant benefits of active treatment, while a larger trial showed a much smaller and a statistically not significant effect (Owen 2004 [53]). These findings suggest that smaller studies demonstrating smaller effects might not have been published.

5. Imprecision

small sample size small number of events

wide confidence intervals uncertainty about magnitude of effect

GRADE From Evidence to Recommendations

Documents

Transcript of GRADE From Evidence to Recommendations