ADOPTING THE GRADE APPROACH FOR CLINICAL PRACTICE GUIDELINES IN GASTROENTEROLOGY – BENEFITS AND

CHALLENGES

AGA Practice Guidelines Committee Meeting, Chicago

May 31, 2009

Yngve Falck-Ytter, M.D.Assistant Professor of Medicine

Case Western Reserve UniversityDirector of Hepatology, VA Medical Center

DisclosureIn the past 5 years, Dr. Falck-Ytter received no personal payments for services from industry. His research group received research grants from Three Rivers, Valeant and Roche that were deposited into non-profit research accounts. He is a member of the GRADE working group which has received funding from various governmental entities in the US and Europe. Some of the GRADE work he has done is supported in part by grant # 1 R13 HS016880-01 from the Agency for Healthcare Research and Quality (AHRQ).

ContentPart 1 Why revisiting guideline

methodology?

GRADE approach Quality of evidence Strength of recommendations

Why societies have adopted GRADE

Content (continued)

Part 2 – practical consideration

Ideal vs. practical ad hoc approaches Funding guideline work

Reassessment of clinical practice guidelines

Editorial by Shaneyfelt and Centor (JAMA 2009) “Too many current guidelines have

become marketing and opinion-based pieces…”

“AHA CPG: 48% of recommendations are based on level C = expert opinion…”

“…clinicians do not use CPG […] greater concern […] some CPG are turned into performance measures…”

“Time has come for CPG development to again be centralized, e.g., AHQR…”

Evidence-based clinical decisions

Research evidence

Patient values and preferences

Clinical state and circumstances

Expertise

Equal for allHaynes et al. 2002

Confidence in evidence

There always is evidence “When there is a question there is

evidence” Evidence alone is never sufficient to

make a clinical decision Better research greater confidence

in the evidence and decisions

Hierarchy of evidence

STUDY DESIGN Randomized Controlled

Trials Cohort Studies and

Case Control Studies Case Reports and Case

Series, Non-systematic observations

BIAS

Expert Opinion

Expert Opinion

Expert Opinion

Reasons for grading evidence?

People draw conclusions about the quality of evidence and strength of

recommendations Systematic and explicit approaches can

help to protect against errors, resolve disagreements communicate information and fulfill needs be transparent about the process

Change practitioner behavior However, wide variation in approaches

GRADE working group. BMJ. 2004 & 2008

10

Which grading system?P: In patients with acute hepatitis C … I : Should anti-viral treatment be used … C: Compared to no treatment …O: To achieve viral clearance?Evidence Recommendation Organization

B Class I AASLD (2009)

VA (2006)II-1 -/-

SIGN (2006)1+ A

AGA (2006)-/- “Most authorities…”

Scenario (2)

Should patients with risk factors for viral hepatitis be screened with a hepatitis C antibody (ELISA) test to identify patients with past hepatitis C exposure?

13

Level of evidence in GI CPGsAASLD AGA ACG ASGE

A Multiple RCTs or meta-analysis

Good Consistent, well-designed, well conducted studies […]

1. Multiple published, well-controlled (?) randomized trials or a well designed systemic (?) meta-analysis

A. RCTs

B Single randomized trial, or non-randomized studies

C Only consensus opinion of experts, case studies, or standard-of-care

Fair Limited by the number, quality or consistency of individual studies […]

Poor … important flaws, gaps in chain of evidence…

2. One quality-published (?) RCT, published well-designed cohort/ case-control studies3. Consensus of authoritative (?) expert opinions based on clinical evidence or from well designed, but uncontrolled or non-rand. clin. trials

B. RCT with important limitations

C. Obser-vational studies

D. Expert opinion

What to do?

14

Limitations of existing systems Confuse quality of evidence with

strength of recommendations Lack well-articulated conceptual

framework Criteria not comprehensive or

transparent GRADE unique

breadth, intensity of development process wide endorsement and use conceptual framework comprehensive, transparent criteria

Focus on all important outcomes related to a specific question and overall quality

Grades of Recommendation Assessment, Development and Evaluation

GRADE Working GroupDavid Atkins, chief medical officera Dana Best, assistant professorb Martin Eccles, professord Francoise Cluzeau, lecturerx

Yngve Falck-Ytter, associate directore Signe Flottorp, researcherf Gordon H Guyatt, professorg Robin T Harbour, quality and information director h Margaret C Haugh, methodologisti David Henry, professorj Suzanne Hill, senior lecturerj Roman Jaeschke, clinical professork Regina Kunx, Associate ProfessorGillian Leng, guidelines programme directorl Alessandro Liberati, professorm Nicola Magrini, directorn

James Mason, professord Philippa Middleton, honorary research fellowo Jacek Mrukowicz, executive directorp Dianne O’Connell, senior epidemiologistq Andrew D Oxman, directorf Bob Phillips, associate fellowr Holger J Schünemann, professorg,s Tessa Tan-Torres Edejer, medical officert David Tovey, Editory

Jane Thomas, Lecturer, UKHelena Varonen, associate editoru Gunn E Vist, researcherf John W Williams Jr, professorv Stephanie Zaza, project directorw

a) Agency for Healthcare Research and Quality, USA b) Children's National Medical Center, USAc) Centers for Disease Control and Prevention, USAd) University of Newcastle upon Tyne, UKe) German Cochrane Centre, Germanyf) Norwegian Centre for Health Services, Norwayg) McMaster University, Canadah) Scottish Intercollegiate Guidelines Network, UKi) Fédération Nationale des Centres de Lutte Contre le Cancer, Francej) University of Newcastle, Australiak) McMaster University, Canadal) National Institute for Clinical Excellence, UKm) Università di Modena e Reggio Emilia, Italyn) Centro per la Valutazione della Efficacia della Assistenza Sanitaria, Italyo) Australasian Cochrane Centre, Australia p) Polish Institute for Evidence Based Medicine, Polandq) The Cancer Council, Australiar) Centre for Evidence-based Medicine, UKs) National Cancer Institute, Italyt) World Health Organisation, Switzerland u) Finnish Medical Society Duodecim, Finland v) Duke University Medical Center, USA w) Centers for Disease Control and Prevention, USAx) University of London, UKY) BMJ Clinical Evidence, UK

GRADE uptake

Where GRADE fits inPrioritize problems, establish panel

Systematic reviewSearches, selection of studies, data collection and

analysisAssess the relative importance of outcomes

Prepare evidence profile: Quality of evidence for each outcome and summary

of findingsAssess overall quality of evidenceDecide direction and strength of recommendation

Draft guidelineConsult with stakeholders and / or external peer

reviewerDisseminate guidelineImplement the guideline and evaluate

GRAD

E

20

GRADE: Quality of evidence

The extent to which our confidence in an estimate of the treatment effect is adequate to support particular recommendation.

Although the degree of confidence is a continuum, we suggest using four categories:

High Moderate Low Very low

I B II V III

Quality of evidence across studies

Outcome #1Outcome #2Outcome #3

Quality: HighQuality: ModerateQuality: Low

Determinants of quality

RCTs start high Observational studies start low What lowers quality of evidence? 5

factors: Detailed design and execution Inconsistency of results Indirectness of evidence Imprecision Publication bias

1. Design and execution

Study limitations (risk of bias)For RCTs: Lack of allocation concealment No true intention to treat principle Inadequate blinding Loss to follow-up Early stopping for benefit

For observational studies: Selection Comparability Exposure/outcome

Schulz KF et al. JAMA 1995 24

Allocation concealment

250 RCTs out of 33 meta-analysesAllocation concealment:Effect

(Ratio of OR)

adequate 1.00 (Ref.)unclear 0.67 [0.60

– 0.75]not adequate 0.59

[0.48 – 0.73]

*

* significant

Cochrane Risk of bias graph in RevMan 5

25

2. Consistency of results

Look for explanation for inconsistency patients, intervention, comparator, outcome,

methods

Judgment variation in size of effect overlap in confidence intervals statistical significance of heterogeneity I2

Pagliaro L et al. Ann Intern Med 1992;117:59-70 27

Heterogeneity

3. Directness of Evidence

Indirect comparisons Interested in head-to-head comparison Drug A versus drug B Tenofovir versus entecavir in hepatitis B

treatment

Differences in patients (early cirrhosis vs end-stage cirrhosis) interventions (CRC screening: flex. sig. vs

colonoscopy) comparator (e.g., differences in dose) outcomes (non-steroidal safety: ulcer on

endoscopy vs symptomatic ulcer complications)

4. Imprecision

Small sample size small number of events wide confidence intervals uncertainty about magnitude of effect

Imprecision

0.75 1.00 1.25

RRappreciable

benefit appreciable harm

imprecise

precise

5. Reporting Bias (Publication Bias)

Reporting of studies publication bias

number of small studies Reporting of outcomes

32

Quality assessment criteria

Lower if…Quality of evidenceHigh (4)

Moderate (3)

Low (2)

Very low (1)

Study limitations(design and execution)

Inconsistency

Indirectness

Imprecision

Publication bias

Observational study

Study designRandomized trial

Higher if…

What can raise the quality of evidence?

BMJ 2003;327:1459–61 33

34

35

Quality assessment criteria

Lower if… Higher if…Quality of evidenceHigh (4)

Moderate (3)

Low (2)

Very low (1)

Study designRandomized trial

Observational study

Study limitations

Inconsistency

Indirectness

Imprecision

Publication bias

Large effect (e.g., RR 0.5)Very large effect (e.g., RR 0.2)

Evidence of dose-response gradient

All plausible confounding would reduce a demonstrated effect

36

Categories of quality

Further research is very unlikely to change our confidence in the estimate of effectHigh

LowFurther research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

ModerateFurther research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

Very low Any estimate of effect is very uncertain

37

Judgments about the overall quality of evidence Most systems not explicit Options:

Benefits Primary outcome Highest Lowest

Beyond the scope of a systematic review GRADE: Based on lowest of all the

critical outcomes

GRADE evidence profile

Going from evidence to recommendations Deliberate separation of quality of

evidence from strength of recommendation

No automatic one-to-one connection as in other grading systems

Example: What if there is high quality evidence, but the balance between benefit and risks are finely balanced?

39

Strength of recommendation

“The strength of a recommendation reflects the extent to which we can, across the range of patients for whom the recommendations are intended, be confident that desirable effects of a management strategy outweigh undesirable effects.”

Although the strength of recommendation is a continuum, we suggest using two categories :

“Strong” and “Weak”

Desirable and undesirable effects Desirable effects

Mortality reduction Improvement in quality of life, fewer

hospitalizations/infections Reduction in the burden of treatment Reduced resource expenditure

Undesirable effects Deleterious impact on morbidity, mortality or

quality of life, increased resource expenditure

4 determinants of the strength of recommendation

Factors that can weaken the strength of a recommendation

Explanation

Lower quality evidence The higher the quality of evidence, the more likely is a strong recommendation.

Uncertainty about the balance of benefits versus harms and burdens

The larger the difference between the desirable and undesirable consequences, the more likely a strong recommendation warranted. The smaller the net benefit and the lower certainty for that benefit, the more likely is a weak recommendation warranted.

Uncertainty or differences in values

The greater the variability in values and preferences, or uncertainty in values and preferences, the more likely weak recommendation warranted.

Uncertainty about whether the net benefits are worth the costs

The higher the costs of an intervention – that is, the more resources consumed – the less likely is a strong recommendation warranted.

Developing recommendations

Implications of a strong recommendation Patients: Most people in this situation

would want the recommended course of action and only a small proportion would not

Clinicians: Most patients should receive the recommended course of action

Policy makers: The recommendation can be adapted as a policy in most situations

Implications of a weak recommendation Patients: The majority of people in this

situation would want the recommended course of action, but many would not

Clinicians: Be prepared to help patients to make a decision that is consistent with their own values/decision aids and shared decision making

Policy makers: There is a need for substantial debate and involvement of stakeholders

6 main misconceptions1. Isn’t GRADE expensive to realize?2. Isn’t GRADE more complicated, takes longer

and requires more resources?3. Isn’t GRADE eliminating the expert ?4. But what about mechanism of disease,

diagnosis, cost?5. But GRADE does not have an “insufficient

evidence to make recommendation” category! (or: the “optional” category), no?

6. But we only “recommend” – we can’t possibly give weak recommendations!

Systematic review

Guideline development

PICO

OutcomeOutcomeOutcomeOutcome

Formulate

question

Rate

importance

Critical

ImportantCritical

Not important

Create

evidence

profile with

GRADEpro

Summary of findings & estimate of effect for each outcome

Rate overall quality of

evidence across outcomes based

on lowest quality of critical outcomes

Panel

RCT start high, obs. data start

low1. Risk of bias2. Inconsisten

cy3. Indirectnes

s4. Imprecision5. Publication

bias

Grad

e

down

Grad

e

up

1. Large effect

2. Dose response

3. Confounders

Rate quality

of evidence

for each

outcomeSelect

outcomes

Very low

LowModerate

High

Formulate recommendations:

• For or against (direction)• Strong or weak (strength)

By considering: Quality of evidence Balance

benefits/harms Values and

preferencesRevise if necessary by considering:

Resource use (cost)

• “We recommend using…”• “We suggest using…”• “We recommend against using…”• “We suggest against using…”

Outcomes

across

studies

Summary, andWhy institutions adopt GRADE1. GRADE is gaining acceptance as international standard

2. GRADE has criteria for evidence assessment across a range of questions and outcomes

3. Criteria for moving from evidence to recommendations

4. Simple, transparent, systematic5. Balance between simplicity and

methodological rigor

Ideal vs. practical ad hoc GRADE approaches

Stage Elements Advantage Comment

Ideal Systematic reviewGRADE eTablesQual. of evidenceStrength of rec.

Follows int. standardsMethodolog. most rigorousEasily maintainableFully transparent process

Access to methodologistAccess to evidence centersInitially more resource

intensive, long-term savings

Inter-mediary

Ad hoc reviewGRADE eTablesQual. of evidenceStrength of rec.

Still retaining major advantages of the of the “ideal approach”

Risk of bias higherAccess methodologist rec.Only minimal addl. cost

Initiation Ad hoc reviewGRADE eTablesQual. of evidenceStrength of rec.

Option to fully “upgrade” to an “ideal approach”

Foundation of a methodo-logically sound system

Risk of bias higherAccess methodologist prnNo additional cost