Extreme Makeover, Data EditionInside the Box

2007 City MatCH Conference – Skills Building Session

Juan M. Acuña M.D., M.Sc.

MCH Medical Epidemiologist

MCHEPI Program Team Leader, CDC

Objectives of the presentation

1. Review validity and the EBPH process

2. Review EPI principles guiding evidence application as a MCH decision driver

3. Review on current examples based on programmatic information in MCH

Relevant questions based on the previous presentation

• Is there a need for EBPH?

• What do we call evidence?

• How do we assess evidence?

• How do we provide recommendations based on the best evidence?

• How do we implement an action plan?

• How do we evaluate impact?

Changes or new programs are based on recommendations

A. Good evidence to support decisions

B. Fair evidence to support decisions

C. Poor evidence that does not provide direction to do or don’t do

D. Fair evidence to support don’t do

E. Good evidence to support don’t do

Let’s change the paradigm…

• MCH covers maternal, child, health

• Other close fields (injury, infections, death, behaviors, etc”

• MCH belongs to public health

• We are temporary MCH practitioners

• Let’s talk about public health

Basic concepts about EBPH

WORKING DEFINITION (PH):

Promoting healthy people in

healthy communities

Public Health• Prevents epidemics and spread of disease

• Protects against environmental hazards

• Prevents injuries

• Promotes and encourages healthy behaviors

• Responds to disasters and assists communities in recovery

• Assures the quality and access to services

APHA

Services

Monitor health status YesDiagnose, investigate public hazards YesInform, educate, empower NoMobilize community partners NoDevelop policies and plans NoEnforce laws and regulations NoLink people to health services NoAssure expert workforce NoEvaluate effectiveness, access, quality YesResearch new insights and solutions yes

Services data-based

Public Health data-action loop:

Case recollection

Population information

Risk factor data (PRAMS)

ANALYSIS

Programs and policies

RATES AND PROPORTIONS

1. absolute risk

2. population “mapping”

3. tendencies

Complex Analyses

• Cause

• risk factors

• costs

• morbidity

Program evaluation

INFORMATION

Inside the box

outside t

he box

Sources of evidence in PH

• “soft” information: review processes, personal information, “gut” feelings

• “adequate” information: routinely collected information, case review programs, passive systems

• “strong” information: active surveillance, some clinical studies

• “very strong”: randomized clinical trials

2. EPI components in public health problems

Examples

• Referral distance is associated to mortality

• Implications of fetal intervention (clinical and public health) to pediatrics and PH

Perinatal Mortality Rate by Distance from Shreveport, Region 7, 1995-9

57.4

18.324.3

61.9

16.5

30.3

9

34.444

010203040506070

0 20 40 60 80 100

Distance from Shreveport

Mo

rtal

ity

Rat

e

p:0.38

r:0.48

Example 1

So the main issue here is Validity

Valid truth

• Unbiased (bias needs to be avoided)• True and tolerable level of chance (chance

needs to be measured)• Took into account the presence of

characteristics that change the relationship of interest (controlled for confounding)

LBW - SGA LAPRAMS data 1998-1999

Population at risk

LA 1998-1999:

130,294 pregnancies

Smoking OR: 3.5

Wt-Gain OR: 3

Counseling OR: 1.7

Prevalence:

LBW: 7% (9,120)

VLBW: 2% (2,605)

SGA: 15% (19,544)

AFp:

LBW: 9% (820)(+?)

VLBW: 2% (52) (+?)

SGA: 2% (390)(+?)

Why the concern?

• Knowledge is rapidly expanding

• The use of “EB decision-making” is common

• Large amount of published (scientific) literature

• Larger amounts of (unused) stored data

• Lack of guidelines for the EBPH process

• Large degree of uncertainty about change

Critical appraisal of the evidence

• The most difficult step (for PH officials)

• Needs technical evaluation (epi)

• Needs “special” skills

• Less than 10% of active personnel has the skills required. Less than 1% has a method.

• Time-consuming step

• Source of “biased” decisions

Critical appraisal of the evidence• Needs assessment,

surveillance (diagnosis):– Prevalence

– Sensitivity

– Specificity

– PPV / NPV

– LRP / LRN

– Pretest odds / postT odds

– post-test probability

– Receiver-operator curve

• Prognosis:– time variables

– survival curves

– time series

– prognosis estimates

– precision

– relationship with screening and diagnosis

Critical appraisal of the evidence

• Intervention:– Blindness (evaluation)

– Randomization (evaluation)

– Control event rate

– Relative Risk reduction

– Absolute Risk reduction

– Number needed to treat

• Risk:– Odds ratio

– Risk ratio

– Incidence

– Prevalence

– Exposure

– inference

– probability distribution

Critical appraisal of the evidence

• Other more “general” terms:– Bias– confounders– probability distribution and chance– p value– confidence intervals– logistic regression, multivariate analysis– univariate analysis

Surveillance Systems

Epidemiological Studies

Prevention Programs

Risk factors

Protective factors

Public concerns

Prevention strategies

Public policy

Education

Prevalence rates

Registry of cases for study or referral

Monitor prevention

Example # 3:

Birth CertificatesPredictive Value Positive 76%Sensitivity 28%

Hospital Discharge DataPredictive Value Positive 85-95%Sensitivity 70-90%

Example # 3:

Evaluation of Data Studies

Conflict in PH

To do things right

To do the right things (right)

DRIVING FORCE: best evidence for the best practice

PROBLEMS: How is this done? How to do it always? How to do it always the same?

Best Evidence

• Makes sense (relevant)

• Unbiased

• Available

• Statistically significant (chance)

• Significant to Public Health

• Leads to correct decisions

Best EvidenceUnbiased: well designed (bias cannot be measured)

random error

average: 10 (very precise) average: 4

Best EvidenceUnbiased: well designed (bias cannot be measured)

Systematic error random error

average: 4 average: 4

Best Evidence

Available:

• Published (strength of evidence)

• Surveillance systems

• Routinely collected information

• Peer information

• Smart opinion

• Other

Evidence

I - Evidence from RCT

II-1 - Well designed non-randomized trials

II-2 - Cohort, Case Control analysis

II-3 - Comparisons of places, time, interventions, better more than 1 center

III - Opinion of authorities, descriptive studies, expert peer groups or committees

Best Evidence

Statistically significant:

Chance (randomness):

•p values

•Confidence intervals

95% p

no association

no difference

Measurable!

Evidence

Statistical significance

Meaningful to Public Health

BOTH

good best fair

We have been taught to accept statistical significance. If large samples (as in many cases), we are bound to have it, even if it is not meaningful.

Best EvidenceLeads to correct decisions: MSAFP vs. Folic Acid for NTDs

• UK and Northern Ireland: one of the highest prevalences in the world

• Prenatal screening for more than 35 years

• Prenatal termination of the affected pregnancies

0

5

10

15

20

25

UK

Ireland

MSAFP

Folic Acid Fortification

Fortificationlevel

Diet + %population> 0.4mg

Casesprevented

% casesprevented

Cases tooccur

0 0 0.6 0 0 2000140 100 2.2 405 20 1595350 250 19.8 919 46 1081420 300 30.6 1082 54 918360 400 48.3 1324 66 676700 500 61.3 1499 75 501

4. Change PH practices

Public Health is about:

• Research

• Advocacy

• Community Services

• Education

• Wisely invest as little money as possible to make the biggest and better change possible

Changes are based on recommendations

A. Good evidence to support decisions

B. Fair evidence to support decisions

C. Poor evidence that does not provide direction to do or don’t do

D. Fair evidence to support don’t do

E. Good evidence to support don’t do

Evidence

I - Evidence from RCT

II-1 - Well designed non-randomized trials

II-2 - Cohort, Case Control analysis

II-3 - Comparisons of places, time, interventions, better more than 1 center

III - Opinion of authorities, descriptive studies, expert peer groups or committees

Framework for evaluation• Do I want (have) to evaluate the study?

• Outline the study

• Is the study believable?

• What is the Public Health relevant finding?– Are the variables of interest included?

– Are findings explainable by chance, bias or confounders?

– Is the finding believable within our knowledge?

• Will the study help me with my population?– Is my population similar?

– Is my problem similar?

– Will the findings benefit my programs and policies?

Inside the Box Workshop Summary

We have:

1. Reviewed concept and situation of “evidence based” practice of public health

2. Reviewed relevant basic EPI concepts

3. Reviewed EPI aspects of public health

4. Provided element to build a framework to evaluate evidence

5. Evaluated the role of evidence in PH decisions