Introduction to evidence-based health science: lessons from medicine

Joseph Lau, MD

September 19, 2011

To be covered

• The arrival of an evidence-based world• Impetus for and a very brief overview of

systematic review (SR) and meta-analysis• An example of a SR to inform pubic health

policy: dietary reference intakes for calcium and vitamin D

• A very brief history and current status of SR in medicine

• ? Lessons to learn for conservation medicine

The Year in IdeasNew York Times Magazine 12/9/2001

Editors and writers included “80 ideas that shook the world (or at least jostled it a little) in 2001 . . .”

Covered: politics, medicine, sports, business, fashion, warfare, . . .

Under medicine: – Evidence-Based Medicine– Pharmacogenomics– Global antiretroviralism– Telesurgery– Turning a bad drug good

Evidence-based health care is the conscientious use of current best evidence in making decisions about the care of individual patients or the delivery of health services. Current best evidence is up-to-date information from relevant, valid research about the effects of different forms of health care, the potential for harm from exposure to particular agents, the accuracy of diagnostic tests, and the predictive power of prognostic factors.

Sackett et al. BMJ 1996

Conventional paradigm for medical practice and education

• See one

• Do one

• Teach one

Google – September 6, 2011

“evidence-based” 22,400,000“evidence-based medicine” 5,000,000“evidence-based practice” 4,030,000“evidence-based nursing” 740,000“evidence-based healthcare” 377,000“evidence-based surgery” 47,400“evidence-based nutrition” 506,000“evidence-based veterinary medicine” 248,000“evidence-based conservation” 121,000“evidence-based ecology” 4,330“evidence-based conservation medicine” 0

“evidence-based management” 5,090,000“evidence-based social” 1,900,000“evidence-based education” 121,000“evidence-based marketing” 505,000“evidence-based politics” 32,900

“clinical practice guideline” 2,560,000”systematic review” 3,210,000“meta-analysis” 18,400,000

The weight of medical knowledgeDavid T. Durack, M.B., D. Phil.; NEJM 1978

Weight of the Index Medicus According to 10-Year Periods from 1879 to 1977

Medical Junk Mail. Montauk L. Lancet 2000;356:344.

Emergency MedicineSan Francisco, USA

Jan 1, 1999 - Dec 31, 1999

122 kg, 2 m high

Lots of studies doesn’t necessarily translate to lots of clinically useful evidence

The Medical Review Articles: State of the Science Mulrow C. Ann Intern Med 1987; 106:485-8.Review of 50 Review Articles in four popular clinical journals over12 months June 1985 - June 1986 (Ann Intern Med, Arch Intern Med, JAMA, NEJM)

Criteria Specified Unclear Not Specified

Purpose 40 1 9

Data Identification 1 2 47

Data Selection 1 0 49

Validity Assessment 1 1 48

Qualitative Synthesis 43 0 7

Quantitative Synthesis 3 1 46

Summary 37 12 1

Future Directive 21 4 25

How health care varies by regionUSA Today, Tuesday, September 19, 2000, 9D

How health care varies by regionUSA Today, Tuesday, September 19, 2000, 9D

Percentage of acute otitis media patients given antibiotics

Figure taken from Froom J et al. Diagnosis and antibiotic treatment of acute otitis media: report from International Primary Care Network. BMJ 1990;300:582-6.

The Importance of Beta, the Type II Error and

Sample Size in the Design and Interpretation of the

Randomized Control Trial

Survey of 71 “Negative” Trials

Freiman JA, Chalmers TC, Smith H Jr, Kuebler R. N Engl J Med 1978;

299:690-4.

Study 1

Study 2

Study 3

Study 4

Cumulative M-A 1

Cumulative M-A 2

Cumulative M-A 3

Pool Studies 1 to 2

Basic Concept of Cumulative Meta-AnalysisStudies ordered

chronologically orby covariates

Pool Studies 1 to 4

Pool Studies 1 to 3

Study n-1 Cumulative M-A n-2

Study n Cumulative M-A n-1 Pool Studies 1 to n

Pool Studies 1 to n-1

Lau J, et al. NEJM 1992

The research of many commentators have already thrown much darkness on this subject, and it is probable that, if they continue, we shall soon know nothing at all about it.

Mark Twain

Summary: why we need systematic reviews and meta-analyses?

• Too much information for practitioners (including domain experts) to keep up and synthesize

• Clinicians and researchers need reliable information; narrative review articles and textbooks often are outdated and biased

• Information of variable quality and reliability, practitioners lack training to evaluate and synthesize research findings

• Studies examined individually offer only partial answers (the whole is greater than the sum of its parts)

• Need for update• Identify research gaps

QUESTION

?

ANSWER

!

Systematic Review

Question refinement

Study identification

Study selection

Data extraction

Quality assessment/

critical appraisal

Data synthesis / meta-analysis

Interpretation

Principles of systematic review

• Prospective research on published data• Reproducible, transparent (clear and thorough

documentation)• Rigorous, take steps to minimize bias and

errors• Involve stakeholders when feasible• When possible, the practice of EBM should be

based on evidence (use practices that are supported by empirical data)

• Caveat: many current methods are evolving

General methods of systematic review• Develop analytic framework• Formulate answerable research question(s) • Specify review criteria

– Participants– Intervention– Comparison– Outcomes

• Identify relevant studies• Take measures to minimize bias; assess potential biases• Synthesize results (qualitative, quantitative [meta-analysis])• Appropriate handling of heterogeneity and interpretation of

results• Identify research gaps

Vitamin D and Calcium: A Systematic Review of Health Outcomes

Tufts Evidence-based Practice Center

August 4, 2009Information Gathering Workshop

Committee to Review DRI for Vitamin D and Calcium

29

Analytic Framework• Relevant questions can be formulated into a

model that analyzes all effects and interactions between intervention or exposure and outcomes; to appreciate relationships of the questions

• Can be used to clarify and generate questions (topics)

• Can highlight what aspects are known and unknown

• Can clarify what study designs may be best to address specific questions

Analytic Framework for Vitamin D and/or Calcium Health Outcomes

Hypertension

BMD, BMC25(OH)D; 1,25(OH)D

Calcium balance

Bone health

CVD

Cancer

Muscle Function

Cancer markers

Blood Pressure

UV exposure

Calcium

Foods & supplements

Arrow 1

Arrow 2

Arrow 2

Arrow 3Arrow 4 Arrow 5

Arrow 5

Arrow 6

Arrow 6

Immune Outcomes

Breast milk or infant circulating levels

Vitamin D2,D3,

25(OH)D

Pregnancy Outcomes

All cause mortality

Body weight, BMI; Growth

Key Question 1. What is the effect of vitamin D, calcium, or combined vitamin D and calcium intakes on clinical outcomes, including growth, cardiovascular diseases, weight outcomes, cancer, immune function, pregnancy or birth outcomes, mortality, fracture, renal outcomes, and soft tissue calcification?

Key Question 2. What is the effect of vitamin D, calcium or combined vitamin D and calcium intakes on surrogate or intermediate outcomes, such as hypertension, blood pressure, and bone mineral density?

Key Question 3. What is the association between serum 25(OH)D concentrations or calcium balance and clinical outcomes? Key Question 4. What is the effect of vitamin D or combined vitamin D and calcium intakes on serum 25(OH)D concentrations?

Key Question 5. What is the association between serum 25(OH)D concentrations and surrogate or intermediate outcomes?

Analytic Framework for Vitamin D and/or Calcium Safety-related (adverse) Outcomes

PICO Selection Criteria

• Population– Generally healthy people with no known disorders– Studies enrolled <20% patients with common diseases allowed– For adverse effects of high intake, any population

• Intervention / Exposure– Observational studies: Serum 25(OH)D or 1,25(OH)2D

• Comparator– Dose relationship

• Outcome– 17 outcomes selected by technical expert panel

• Study Design– Experimental / Observational, Duration, N; (excluded cross-

sectional studies and retrospective case-control)

Citations identified in MEDLINE and CochraneCentral database search for primary studies,

published between 1969 and Arpil 2009(n=16,733)

Citations identified in MEDLINE, CochraneDatabase of Systemic Reviews, and the HealthTechnology Assessments database search for

systematic review articles published beforeDecember, 2008 (n=1,746)

Primary study articles retreived for full-text review(n=584)

Systematic review articles retreived for full-textreview (n=68)

Abstracts failed to meetcriteria (n=17,825)

Primary study articles reviewed (n=165)- 60 randomized, controlled trials

- 3 nonrandomized comparative studies- 102 observational studies (either cohort or

nested case-control studies)

Systematic reviews included (n=11)

Articles failed to meetcriteria (n= 476)

Literature search and selection

Critical Appraisal of Primary Studies

We adapted a 3-category grading system of the AHRQ Methods Reference Guide for Comparative Effectiveness Reviews, which encompasses the principles of CONSORT statement for RCTs, STROBE checklist for observational studies. This system defines a generic grading system that is applicable to each type of study design.

A - Least bias; results are valid

B - Susceptible to some bias, but not sufficient to

invalidate the results

C - Significant bias that may invalidate the results

Critical Appraisal of Systematic Reviews

• A summary quality grade for systematic review is difficult to interpret

• Various dimensions and nuances of the systematic review must be understood

• We applied AMSTAR checklist, a tool to assess the quality of reporting of systematic reviews

• Items evaluated are made explicit for the reader• Comments provided where appropriate

Reporting of Evidence

• Evidence tables – detailed information about each study

• Summary tables– summary from each study that address a

question (outcome, study design)

• Figures, graphs• Meta-analyses (if appropriate)• Narratives, highlight features and

limitation of study in answering question

Results

Primary Studies on Vitamin D Intake or Concentration

Availability and Quality of Primary Studies

Quality grade of primary studies

A (11%) B (50%) C (39%)

Vitamin D (94)

RCT (30) 3 14 13

Non-RCT (64) 4 33 27

Calcium (108)

RCT (23) 1 11 11

Non-RCT (85) 17 41 27

Vitamin D + Calcium (19)

RCT (19) 0 12 7

Non-RCT (0) 0

Total = 221 (from 165 unique studies); same study may contribute more than one outcome

Organization of Results Chapter

• Vitamin D– outcomes

• Calcium– outcomes

• Vitamin D + calcium– Outcomes

• Association with intake and serum Vitamin D level (Question 4)

• Adverse or Safety Outcomes

Reporting of Individual Outcome

• Synopsis• Detailed presentation (highlight features and

limitation of study in answering question)– Findings per vitamin D concentration– Findings per age and sex– Findings by life stage

• Summary tables– RCTs

• Characteristics• Results

– Cohort studies• Characteristics • Results

• Figures, graphs• Meta-analyses (if appropriate)

Vitamin D and colorectal cancer: results of RCTs (pg 68)

Vitamin D and colorectal cancer: results of RCTs (pg 70)

Vitamin D and colorectal cancer – observational studies (pg 74)

Vitamin D and colorectal cancer: Results of observational studies (pg 76)

Caveats• YGWYA (you get what you asked) - report

provide information only on questions formulated a priori, it does not answer all potential questions of interest

• As much as we strive to be objective, some judgment is involved (grading of studies)

• No standard tool available to assess quality of nutritional observational studies

• Publication bias• Need to rely on existing systematic reviews,

but using them is challenging• Studies in general were not designed for DRI

issues (e.g., targeting DRI life stages)

A very brief history and current status of SR in medicine / healthcare

• First meta-analysis• Current era (starting in late 1970s)• Criticisms • Methods advancements • EBM era• US government effort (AHRQ EPC)• Healthcare reform (comparative

effectiveness reviews)

Methodological advances• Empirical research

– Validity of meta-analysis– Large studies vs meta-analysis of smaller studies– Understanding of heterogeneity

• Statistical methods• Computer software• Questions formulation / analytic framework• Searching literature• Improving quality of primary studies• Interpretation and using of results• Future research needs

Applications of systematic reviews and meta-analyses in healthcare / biomedical research

• Interventions (many 10,000’s)• Epidemiologic (several 1,000’s)• Diagnostic test accuracy (~1,000)• Genetic tests / Genomics (~2,000)• Health economics (<100)• Basic science studies (<100)• All share similar principles; unique issues and

methods required for each category

CommunityBased

Stakeholders

Translational Spectrum of Comparative Effectiveness Research

Academic MedicalCenters

EvidencePrioritization

(Qualitative / Quantitative Elicitation)

EvidenceGeneration

(Randomized Control Trials,Registries, Observations,

N-of-Trials)

EvidenceGeneration

(Randomized Control Trials,Registries, Observations,

N-of-Trials)

EvidenceSynthesis

(Systematic Reviews, Meta-Analysis)

EvidenceInterpretation

and Integration

(Cost-Effectiveness AnalysisDecision Analysis)

Dissemination and Application(Guidelines, Policy,

Social Sciences,Implementation Science)

Feedback andAssessment

(Qualitative Elicitation, Data Monitoring, Quality

Monitoring / Measurement)

Tufts CTSI