1

EPI235: Epi Methods in HSR

April 17, 2007 L5

Program Evaluation with Longitudinal Data 1: Applications (Dr. Schneeweiss)

Various examples of applications in Health Services Research. Strengths and limitations of time series analysis .

Background reading: •Soumerai SB, Avorn J, Gortmaker S, Ross‑Degnan D. Payment restrictions for prescription drugs in Medicaid: Effects on therapy, cost, and equity. New Engl J Med 1987;317:550‑556.•Soumerai SB, Ross-Degnan D, Avorn J, McLaughlin TJ. Effects of Medicaid drug-payment limits on admission to hospitals and nursing homes. N Engl J Med 1991;325:1072-1077

2

3

4

5

What is Program Evaluation?

6

Program evaluation

What is program evaluation?

Design options:

Randomized trial

Cohort study

Quasi-experiment

What means quasi-experiment?

- implementation vs. evaluation perspective

- unintended randomization

Quasi-experiment = pre-post comparison?

= longitudinal design

= repeated measure design

Program Evaluation

7

8

9

10

11

12

Observational Design: Simple pre-post comparisons

Time

Intervention

Assumptions for causal inference:

1. The pre experience represents the post experience had there been no intervention

Outcome rate

13

Observational Design: Simple pre-post comparisons

Time

Intervention

Threat to causal inference:

Single pre-post estimates are averages of an underlying trend independent of the Intervention

Outcome rate

14

25% cost sharing in Quebec

0.0

0.7

1.4

2.1

# o

f e

ss

en

tia

l dru

gs

pe

r d

ay

ObservedPredicted (--- 95% CI)

11-93 8-94 8-95 8-96 8-97

Elderly (n=70801)

Policy Initiated

Tamblyn, JAMA 2000

15

Observational Design: no concurrent controls

Time

Intervention

Assumptions for causal inference:

1. Close temporal relation

2. Extrapolation of baseline trend is equal to the counterfactual experienceSchneeweiss, Health Policy 2000

16

Fixed ¢50 cost sharing in SC

Nelson, Med Care 1984

17

Observational Design: With concurrent controls

Assumptions for causal inference:

1. …

2. Control trend is equal to the counterfactual experience of intervention group

Time

Intervention

Intervention group

Control group

Schneeweiss, J Clin Epi 2002

18

3 prescription caps in NH:

80%

85%

90%

95%

100%

Baseline Cap After Cap

% o

uts

ide

nu

rsin

g h

om

e

New Jersey

New Hampshire

Soumerai NEJM 1991

19

Longitudinal quasi-experimental evaluation designs

Strength Weaknesses

Controls for between-person

confounding

Immediate causal inference

Intuitive graphs – easy to understand

Violation of basic assumption

Co-intervention or long-term trend

Slow or delayed effect

Regression to the mean

Reactiveness

Correlated data

Within-person confounding?

20

… or randomization?

Time

Intervention

Intervention group

Control group

R

Assumptions for causal inference:

1. Subjects comply with their assigned ‘treatment’ = policy

21

Complex statistical analyses are less convincing for decision makers

22

Formulary delisting in BC

23

More Examples:

24

More Examples:

Prescription drug useSurgical site infectionsContraindicated drug use

25

Medicaid prior authorization: Use

Smalley NEJM 1995

26

Medicaid prior authorization: $$

Smalley NEJM 1995

27

Open squares = utilization indicator (cesarean sections receiving perioperative antibiotic prophylaxis) Open circles = timing indicator (antibiotic within 1 hour of delivery)Solid diamonds = surgical site infection rate after cesarean section

Period 1 was a baseline period. Periods 2 and 3 were successive intervention periods.

Weinberg et al.: Arch Intern Med 2001

Reducing surgical site infections after C-section: Hospital A

28

Open squares = utilization indicator (cesarean sections receiving perioperative antibiotic prophylaxis) Open circles = timing indicator (antibiotic within 1 hour of delivery)Solid diamonds = surgical site infection rate after cesarean section

Period 1 was a baseline period. Periods 2 and 3 were successive intervention periods.

Weinberg et al.: Arch Intern Med 2001

Reducing surgical site infections after C-section: Hospital B

29

The Intervention:

'Dear Doctor' letters concerning interactions between cisapride and a series of drugs.

A letter in 1995 described a risk of prolonged QT intervals and serious ventricular arrhythmia in patients who received macrolide antibiotics and imidazole antifungals in conjunction with cisapride.

A June 1998 letter that expanded the list of contraindicated comedications had wider distribution than an earlier one, was accompanied by substantial Internet and media coverage, and was complemented by an effort to inform large pharmacy dispensing information organizations of the warnings against concurrent use of the named drugs.

FDA Risk Management: Effectiveness of a ‘Dear Doctor’ letter

30

Weatherby LB et al. PharmEpi Drug Safety 2001

31

32

Lecture on LPUs:

Weatherby LB et al. Clin Pharm Ther 2002

33

The Policy Model

34

Policy Model: reference pricing

35

The Clinical Model

36

More on the Clinical Model:

37

Clinical Model: reference pricing

38

Policy Model vs. Clinical Model