Observational Study Designs and

Studies of Medical Tests

Tom Newman August 17, 2010

Thanks to Michael Kohn

Outline Conceptual overview Review common observational study designs

Cohort, Double Cohort Case-Control Cross-sectional

Studies of Medical Tests Diagnostic Test Accuracy Prognostic Test Accuracy

Examples “Name that study”

Caveats Nomenclature is confusing and used

inconsistently “Cross-sectional” can refer to timing or sampling “Retrospective” does not always mean

retrospective Getting the name right is helpful, but it is

more important to be able to explain what you want to do and have it make sense for your RQ

If you can’t name your study it’s worth making sure it makes sense

Key elements of study design

Timing of the study Timing of variable occurrence and

measurement How the subjects will be sampled

Timing of the study Prospective: investigator enrolls

subjects and makes measurements in the present and future

Historical: investigator relates predictor variables that have already been measured to outcomes that have already occurred

Retrospective: can mean historical, but best reserved for case-control studies

Prospective studies

Control over subject selection and variable measurements

Have to wait for outcomes to occur Take longer More expensive

Historical studies

Less control over subject selection and variable measurements

Outcomes have already occurred Done sooner Less expensive

Timing of measurements

Longitudinal: measurements in subjects made at more than one time

Cross-sectional: predictor and outcome measured at the same time

Longitudinal timing of measurements Predictor variable precedes outcome

Better for causality (reduces likelihood of “effect-cause”)

Measurement of predictor precedes measurement of outcome No need for blinding of measurement of

predictor variable Needed to measure incidence = new

cases/population at risk/time Risk of getting the disease

Cross-sectional timing of measurements Measurement of predictor and outcome

at about the same time Causality may be more difficult to infer No loss to follow-up

Can only measure prevalence = existing cases at one point in time/population at risk Prevalence = incidence x duration Risk of having the disease Not as good for causality

Example: “Incidence-Prevalence Bias”

In asymptomatic adults, prevalence of coronary calcium is lower in blacks than in whites*

Does this mean blacks get less heart disease?

No, incidence is greater, but duration is shorter**

*Doherty TM et al J Am Coll Cardiol. 1999;34:787–794

**Nieto FJ, Blumenthal RS. J Am Coll Cardiol, 2000; 36:308-309

Sampling of subjects By predictor variable By outcome variable By other (e.g., demographic) factors that

define the population of interest Sometimes called “cross-sectional” sampling

Usually best

Study designs

Descriptive Many studies of medical tests Hint variables must VARY

If either the predictor or outcome variable does not vary in your study (e.g., because one value is an inclusion criterion) your study is descriptive

Analytical

Analytical study designs

Experimental-- Randomized trial

Observational (today’s topic)-- Cohort -- Double Cohort (exposed-unexposed)-- Case-control-- Cross-sectional

Observational analytic studies

Causality is important May be only ethical option for studying

risk factors for disease Often more efficient Populations may be more representative More intellectually interesting than RCTs?

Note on Figures

Following schematics of observational study designs assume:

Predictor = Risk Factor Outcome = Disease Both dichotomous

Cohort Study

Prospective Cohort Study

Historical Cohort Study

THE PAST

Cohort Studies

1) Measure predictor variables on a sample from a population (defined by something other than the variables you are studying).

2) Exclude any subjects who already have the outcome.

3) Follow the subjects over time and attempt to determine outcome on all subjects.

Cohort Studies are longitudinal

Can identify individuals lost to follow up

Can estimate the incidence of the outcome in the population (e.g., cases/person-year)

Measure of disease association is the relative risk (RR) or relative hazard (RH)

Double Cohort Study

Double Cohort (Exposed-Unexposed) Studies

1. Sample study subjects separately based on predictor variable

2. Exclude potential subjects in whom outcome has already occurred.

3. Attempt to determine outcome in all subjects in both samples over time.

Double Cohort (Exposed-Unexposed) Studies

Can identify individuals lost to follow up Can measure incidence in each cohort,

but not overall incidence in the population*

Measure of disease association is the relative risk (RR) or relative hazard (RH)

*Unless one of the cohorts is a sample of everyone not in the other cohort

Cohort Studies: Summary Timing of the STUDY

Prospective Historical

Timing of the MEASUREMENTS: All cohort studies are longitudinal (follow

patients over time) SAMPLING

Cohort study – sample based on other (e.g., demographic) characteristics

Double cohort study -- sample on predictor variable

Case-Control Study

Case-Control Study

1) Separately sample subjects with the outcome (cases) and without the outcome (controls)

2) Attempt to determine predictor status on all subjects in both outcome groups

Case-Control Study Cannot identify individuals lost to follow up

(no such thing as “lost to follow up”, since by definition outcome status is known)

Cannot calculate prevalence (or incidence) of outcome

Measure of disease association is the Odds Ratio (OR)

Try to replicate a nested case control study in which the cases and controls arise from the same cohort.

Nested Case-Control Study

Cross-Sectional Study

Cross-Sectional Study

Attempt to determine predictor and outcome status on all patients in a single population (defined by something other than predictor or outcome).

Cross-Sectional Study

No loss to follow-up Can calculate prevalence but not

incidence Measure of disease association is the

Relative Prevalence (RP). Can be prospective or historical

Eliminate subjects who already have disease

Cohort Studies Start with a Cross-Sectional Study

Studies of Medical Tests Causality often irrelevant. Not enough to show that test

result is associated with disease status or outcome*.

Need to estimate parameters (e.g., sensitivity and specificity) describing test performance.

*Although if it isn’t, you can stop.

Studies of Diagnostic Test Accuracy for Prevalent Disease

Predictor = Test ResultOutcome = Disease status as

determined by Gold Standard

Designs:

Case-control (sample separately from disease positive and disease negative groups)

Cross-sectional (sample from the whole population of interest)

Double-cohort-like sampling (sample separately from test-positive and test-negative groups)

Dichotomous TestsDisease + Disease -

Test +a

True Positivesb

False Positives

Test -c

False Negatives

dTrue

Negatives

Total

a + cTotal With

Disease

b + dTotal

WithoutDisease

Sensitivity = a/(a + c)Specificity = d/(b + d)

Studies of Dx Tests

Importance of Sampling Scheme

If sampling separately from Disease+ and Disease– groups (case-control sampling), cannot calculate prevalence, positive predictive value, or negative predictive value.

Dx Test: Case-Control Sampling

Disease +Sampled

Separately

Disease –Sampled

Separately

Test +a

True Positives

bFalse Positives

Test -c

False Negatives

dTrue Negatives

Totala + c

Total With Disease

b + dTotal Without

Disease

Sensitivity = a/(a + c) Specificity = d/(b + d)

Dx Test: Cross-sectional Sampling

PPV = a/(a + b)

NPV = d/(c + d)

Prevalence = (a + c)/N

Disease + Disease - Total

Test +

aTrue

Positives

bFalse

Positives

a + bTotal

Positives

Test -

cFalse

Negatives

dTrue

Negatives

c + dTotal

Negatives

Total a + cTotal With

Disease

b + dTotal

WithoutDisease

a + b + c + d

Total N

Studies of Prognostic Tests for Incident Outcomes

Predictor = Test ResultDevelopment of outcome or time to

development of outcome.

Design: Cohort study

Examples

Name that observational study design

Babies born at Kaiser with severe neonatal hyperbilirubinemia (Bili 25) were compared with randomly selected “controls” from the same birth cohort.

Outcomes: (blinded) IQ test and neurologic examination at age 5 years.

Results: No difference in IQ or fraction with neurologic disability between the “case” and “control” groups.

Newman, T. B., P. Liljestrand, et al. (2006). N Engl J Med 354(18): 1889-900.

Jaundice and Infant Feeding Study

Design?(Be Careful)

JIFeeDouble Cohort (Exposed-Unexposed) Study*

The subjects are divided by predictor (Bili 25+), not outcome (neurologic disability). The “cases” are actually the exposed group and the “controls” are actually the unexposed group

*Actually a nested triple cohort study, since “cases” and “controls” came from the same birth cohort and we also studied dehydration. See Hulley page 104.

HIV Tropism and Rapid Progression*

* Vivek Jain’s Project

Is HIV CXCR4 (as opposed to CCR5) tropism a predictor of rapid progression in acutely infected HIV patients?

Molecular tropism assay is expensive. Have funding to perform a total of 80 assays.

UCSF OPTIONS cohort follows patients acutely infected with HIV. Has banked serum from near time of acute infection.

HIV Tropism and Rapid Progression (continued)

Identify the 40 patients with the most rapid progression (Group 1) and randomly select 40 others from the UCSF Options cohort (Group 2).

Run the tropism assay on banked serum for these 80 patients and compare results between Group 1 and Group 2.

Design?

HIV Tropism and Rapid Progression

Nested Case-Control Study

HIV Tropism and Rapid Progression

RRISK(Reproductive Risk Factors for Incontinence at Kaiser)

Random sample of 2100 women aged 40-69 years old

Interview, self report, diaries to determine whether they have the outcome, urinary incontinence.

Chart abstraction of obstetrical and surgical records to establish predictor status

RRISK

Design?

RRISK Funded with an R01 by the NIDDK as a

retrospective cohort study Longitudinal, but can’t tell loss to follow-up,

incidence of incontinence, or relative risk from this design

Michael calls it a cross-sectional study Tells us prevalence of incontinence But not all measurements made at the

same time It’s a lot like a nested case control study

But did not employ “case-control” sampling Nested cross-sectional study?

Steroid treatment in the ED and among children hospitalized for asthma

Research Question: what are the frequency and predictors of delayed receipt of steroids in the ED among children admitted for asthma?

Subjects: children admitted for asthma Predictors: age, time of arrival, etc. Outcome: Time to steroid

administration

Steroid treatment in the ED and among children hospitalized for asthma -2

This study is hard to name Time to event data makes this sound like a

cohort study (even if follow-up time is very short) Define a group at risk of the outcome Measure predictors Follow for outcome occurrence

But there is a problem: You can’t define a cohort based on variables not

present at baseline

Steroid treatment in the ED and among children hospitalized for asthma -3

Possible changes Make it a descriptive study of hospitalized

patients Make it a cohort study of ED patients

Could study predictors of time to steroids Time to steroids could be a predictor of

hospitalization

Association of lipid‑laden alveolar macrophages (LLAM) and gastroesophageal reflux (GER) in children*

Did pH probe, barium swallow, and endoscopy on 115 children with chronic respiratory tract disorders to determine GE reflux Group 1: 74 children with GER Group 2: 41 children with no GER

Bronchoscopy and bronchial lavage to determine LLAM LLAM were present in 63/74 (85%) with GER LLAM in 8/41 (19%) children without GER P < 0.0001

Design?

*J Pediatr 1987;110:190‑4

Association of lipid‑laden alveolar macrophages (LLAM) and gastroesophageal reflux (GER) in children*

Design:

Cross-sectional study of diagnostic test accuracy (with cross-sectional sampling)

*J Pediatr 1987;110:190‑4

Association of lipid‑laden alveolar macrophages and gastroesophageal reflux in children -3

J Pediatr 1987;110:190‑4

Conclusions: “We suggest that LLAM from bronchial lavage may be a useful marker for tracheal aspiration in children with GER in whom chronic lung disease may subsequently develop.”

What is wrong?

Association of lipid‑laden alveolar macrophages and gastroesophageal reflux in children -4

J Pediatr 1987;110:190‑4

Conclusions: “We suggest that LLAM from bronchial lavage may be a useful marker for tracheal aspiration in children with GER in whom chronic lung disease may subsequently develop.”

Study design does not permit this conclusion Can’t estimate risk of developing lung disease from

a Cross-sectional sample that Includes only patients with lung disease