© 2003 By Default! A Free sample background from Slide 1 Methodologies and Automated Applications...

© 2003 By Default!

A Free sample background from www.powerpointbackgrounds.com

Slide 1

Methodologies and AutomatedMethodologies and Automated Applications for Post-Marketing Applications for Post-Marketing

Outcomes Surveillance of Medical Outcomes Surveillance of Medical Devices and MedicationsDevices and Medications

Michael E. Matheny, MD, MSMichael E. Matheny, MD, MS

NLM Biomedical Informatics FellowNLM Biomedical Informatics Fellow

Decision Systems Group, Department of Radiology Decision Systems Group, Department of Radiology Brigham & Women’s Hospital, Boston, MABrigham & Women’s Hospital, Boston, MA

© 2003 By Default!


Slide 2

OutlineOutline

Post-Marketing Surveillance BackgroundPost-Marketing Surveillance Background Statistical Methodology DevelopmentStatistical Methodology Development Computer Application DevelopmentComputer Application Development Clinical ExamplesClinical Examples Future Directions Future Directions

© 2003 By Default!


Slide 3

BackgroundBackgroundSurveillance RationaleSurveillance Rationale

Phase 3 Trials insufficient to ensure Phase 3 Trials insufficient to ensure adequate safety of medications and devicesadequate safety of medications and devices

– Low frequency events are not detectedLow frequency events are not detected

– Protected populations (pregnant women, Protected populations (pregnant women, children) and more ill populations not children) and more ill populations not representedrepresented

– Complications delayed by a number of years Complications delayed by a number of years cannot be detectedcannot be detected

© 2003 By Default!


Slide 4

BackgroundBackgroundFDA Medical DevicesFDA Medical Devices

1,700 types of devices1,700 types of devices

500,000 device models500,000 device models

23,000 manufacturers23,000 manufacturers

© 2003 By Default!


Slide 5

BackgroundBackgroundFDA New Drug Applications (NDA)FDA New Drug Applications (NDA)

0

20

40

60

80

100

120

140

Applications

Approvals

© 2003 By Default!


Slide 6

BackgroundBackgroundCurrent Post-Marketing SurveillanceCurrent Post-Marketing Surveillance

Combination of mandatory and voluntary Combination of mandatory and voluntary adverse event reportingadverse event reporting

– Mandatory reporting by manufacturers and health Mandatory reporting by manufacturers and health facilitiesfacilities

– Voluntary MedWatch / MAUDE reports by providers and Voluntary MedWatch / MAUDE reports by providers and patientspatients

2004 Drug-Related Adverse Event Reports2004 Drug-Related Adverse Event ReportsTotalTotal 422,889422,889 Manufacturer & Facility ReportsManufacturer & Facility Reports 401,396401,396 MedWatchMedWatch 21,49321,493

© 2003 By Default!


Slide 7

0.173%

0.131%

0.102%

0.045%

0.234%

0.198%

0.116%

0.00%

0.05%

0.10%

0.15%

0.20%

0.25%

0.30%

Q2 Q3 Q4 Q1 Q2 Q3 Q4

0.0%

BackgroundBackgroundMAUDE Cypher Reporting RateMAUDE Cypher Reporting Rate

2003 2004

FDA Warning FDA Warning Cancelled

© 2003 By Default!


Slide 8

BackgroundBackgroundCurrent Post-Marketing SurveillanceCurrent Post-Marketing Surveillance

‘‘Phase 4’ TrialsPhase 4’ Trials

– Poor CompliancePoor Compliance

• As of March 2006 report, 797 of 1231 (65%) agreed-As of March 2006 report, 797 of 1231 (65%) agreed-upon trials had yet to be startedupon trials had yet to be started

– BarriersBarriers

• Lack of manufacturer incentivesLack of manufacturer incentives– ExpensiveExpensive– Drug already on the marketDrug already on the market

• Lack of regulatory enforcementLack of regulatory enforcement

© 2003 By Default!


Slide 9

BackgroundBackgroundMedical Device RecallsMedical Device Recalls

Boston Scientific cardiac stent (1998)Boston Scientific cardiac stent (1998)– Balloon rupture at low pressuresBalloon rupture at low pressures

Guidant cardio-defibrillator (2005)Guidant cardio-defibrillator (2005)– Malfunction due to electrical shortMalfunction due to electrical short

Vioxx (2004)Vioxx (2004)– cardiovascular complicationscardiovascular complications

Tequin (2006)Tequin (2006)– Hypoglycemia and hyperglycemiaHypoglycemia and hyperglycemia

© 2003 By Default!


Slide 10

BackgroundBackgroundFDA ResponseFDA Response

Increasing demands for Phase 4 trialsIncreasing demands for Phase 4 trials

Legislation to increase quality of adverse Legislation to increase quality of adverse event reportingevent reporting

Emphasizing trial registries (clinicaltrials.gov) Emphasizing trial registries (clinicaltrials.gov) as way to prevent omission of resultsas way to prevent omission of results

Commissioned IOM report “The Future of Commissioned IOM report “The Future of Drug Safety”Drug Safety”

© 2003 By Default!


Slide 11

BackgroundBackgroundAdverse Event Data ContinuumAdverse Event Data Continuum

Phase 3Trials

MAUDE /MedWatch

VoluntaryRegistry

MandatoryRegistry

Internal Validity +++ -- ++ +++

Generalizibility +/- +/- ++ +++

Breadth ++ +++ ++ ++

Immediacy --- +++ ++ ++

Lack of Bias +++ --- --- ++

© 2003 By Default!


Slide 12

Statistical MethodsStatistical MethodsMedical Outcomes MonitoringMedical Outcomes Monitoring

Using registry data that tracks all patients Using registry data that tracks all patients allows different types of analysis than used in allows different types of analysis than used in the FDA’s adverse event reporting systemsthe FDA’s adverse event reporting systems

No generally accepted methods for No generally accepted methods for monitoring registry data for adverse eventsmonitoring registry data for adverse events

– Lack of sufficient discrete electronic data Lack of sufficient discrete electronic data sources to construct registriessources to construct registries

– Some outcomes are challenging or expensive to Some outcomes are challenging or expensive to track for an entire populationtrack for an entire population

© 2003 By Default!


Slide 13

ObjectiveObjective

Develop methodologies and implement an Develop methodologies and implement an automated computer monitoring system to automated computer monitoring system to perform outcomes surveillance of registry perform outcomes surveillance of registry data for new medical devices and data for new medical devices and medicationsmedications

© 2003 By Default!


Slide 14

Statistical MethodsStatistical MethodsStatistical Process ControlStatistical Process Control

Period

0 2 4 6 8 10 12 14

Eve

nt P

ropo

rtio

n

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

0.20

© 2003 By Default!


Slide 15

Statistical MethodsStatistical MethodsBayesian Updating StatisticsBayesian Updating Statistics

© 2003 By Default!


Slide 16

Statistical MethodsStatistical MethodsEstablishing Baseline DataEstablishing Baseline Data

Primary Data SourcesPrimary Data Sources

– Phase 3 trial dataPhase 3 trial data

– Post-Marketing data from a closely related Post-Marketing data from a closely related medication/devicemedication/device

Alternative Data SourcesAlternative Data Sources

© 2003 By Default!


Slide 17

Statistical MethodsStatistical MethodsEstablishing Alerting ThresholdsEstablishing Alerting Thresholds

Use number of events and sample size to calculate Use number of events and sample size to calculate proportion with confidence intervalsproportion with confidence intervals

Typically, medical domains use 95% CI or 1.96 Typically, medical domains use 95% CI or 1.96 sigma from the point estimatesigma from the point estimate

© 2003 By Default!


Slide 18

Statistical MethodsStatistical Methods Establishing Alerting ThresholdsEstablishing Alerting Thresholds

SPCSPC

BUSBUS

© 2003 By Default!


Slide 19


© 2003 By Default!


Slide 20


Wilson’s method of comparison between two Wilson’s method of comparison between two proportionsproportions

© 2003 By Default!


Slide 21

Statistical MethodsStatistical MethodsRisk StratificationRisk Stratification

Allows creating subgroups for separate Allows creating subgroups for separate analysesanalyses

Single variableSingle variable

Logistic regression model with scoring Logistic regression model with scoring thresholdsthresholds

© 2003 By Default!


Slide 22

Application DevelopmentApplication DevelopmentDELTADELTA

Data Extraction and Longitudinal Time Data Extraction and Longitudinal Time Analysis (DELTA)Analysis (DELTA)

Design GoalsDesign Goals– Generic data import formatGeneric data import format– Allow both prospective and retrospective Allow both prospective and retrospective

analysesanalyses– Modular framework to allow sequential addition Modular framework to allow sequential addition

of statistical methodologiesof statistical methodologies– Multiple alerting methodsMultiple alerting methods– Any number of concurrent ongoing analysesAny number of concurrent ongoing analyses

© 2003 By Default!


Slide 23


Data Dictionary Statistical Modules

DELTA Database

Clinical Data Entry

Source Database

Source IT Manager

SPC

BUSVP

NIn

tra

ne

t

DELTA Users

WebServer

© 2003 By Default!


Slide 24

Application Test DataApplication Test DataCypher Drug-Eluting Stent (DES)Cypher Drug-Eluting Stent (DES)

Setting:Setting:– Brigham & Women’s Hospital Brigham & Women’s Hospital (07/2003 – 12/2004)(07/2003 – 12/2004)

Population:Population:– All patients receiving a drug-eluting stent (2270)All patients receiving a drug-eluting stent (2270)

Outcome:Outcome:– Post-procedural in-hospital mortality (27)Post-procedural in-hospital mortality (27)

Baseline:Baseline:– University of Michigan Data (1997-1999)University of Michigan Data (1997-1999)

© 2003 By Default!


Slide 25


© 2003 By Default!


Slide 26


© 2003 By Default!


Slide 27


© 2003 By Default!


Slide 28


© 2003 By Default!


Slide 29


© 2003 By Default!


Slide 30


© 2003 By Default!


Slide 31


© 2003 By Default!


Slide 32


© 2003 By Default!


Slide 33


© 2003 By Default!


Slide 34

Risk StratificationRisk StratificationPotential SolutionPotential Solution

Incorporate individual risk prediction models Incorporate individual risk prediction models in order to adjust for case mix and illness in order to adjust for case mix and illness severityseverity

© 2003 By Default!


Slide 35

Possible Risk Prediction MethodsPossible Risk Prediction Methods

Linear / Logistic RegressionLinear / Logistic Regression

Artificial Neural NetworksArtificial Neural Networks

Bayesian NetworksBayesian Networks

Support Vector MachinesSupport Vector Machines

© 2003 By Default!


Slide 36

LR External Validation LR External Validation ModelsModels

Model Dates Location Sample

NY 1992 1991 NY 5827

NY 1997 1991 – 1994 NY 62670

CC 1997 1993 – 1994 Cleveland, OH 12985

NNE 1999 1994 – 1996 NH, ME, MA, VT 15331

MI 2001 1999 – 2000 Detroit, MI 10796

BWH 2001 1997 – 1999 Boston, MA 2804

ACC 2002 1998 – 2000 National 100253

© 2003 By Default!


Slide 37

LR External ValidationLR External Validation


Population:Population:– All patients undergoing percutaneous coronary All patients undergoing percutaneous coronary

intervention (5216)intervention (5216)

Outcome:Outcome:– Post-procedural in-hospital mortality (71)Post-procedural in-hospital mortality (71)

© 2003 By Default!


Slide 38

LR External Validation LR External Validation ResultsResults

Curve Deaths AUC HL χ2 HL (p)

NY 1992 96.7 0.82 31.1 <0.001

NY 1997 61.6 0.88 32.2 <0.001

CC 1997 78.8 0.88 27.8 <0.001

NNE 1999 56.2 0.89 45.9 <0.001

MI 2001 61.8 0.86 30.4 <0.001

BWH 2001 136.1 0.89 39.7 <0.001

ACC 2002 49.9 0.90 42.0 <0.001

BWH 2004 70.5 0.93 7.61 0.473

© 2003 By Default!


Slide 39

LR External Validation LR External Validation ConclusionsConclusions

Excellent discrimination across all modelsExcellent discrimination across all models

Calibration (Hosmer-Lemeshow) poor for all Calibration (Hosmer-Lemeshow) poor for all models but recent local onemodels but recent local one

Addressed categorical risk stratification by Addressed categorical risk stratification by keeping all records in one stratumkeeping all records in one stratum

Calibration problems over time limit Calibration problems over time limit application, and require exploration of application, and require exploration of recalibration methodsrecalibration methods

© 2003 By Default!


Slide 40

OPUS (TIMI-16)OPUS (TIMI-16)

Setting:Setting:– 888 Hospitals in 27 Countries888 Hospitals in 27 Countries

Intervention:Intervention:– Oral IIb-IIIa Inhibitor vs PlaceboOral IIb-IIIa Inhibitor vs Placebo

Population:Population:– Intervention Arm [Both arms identical at 30 days] (6867)Intervention Arm [Both arms identical at 30 days] (6867)

Outcome:Outcome:– 30 day mortality30 day mortality– Trial stopped early due to elevation in intervention armTrial stopped early due to elevation in intervention arm

Baseline:Baseline:– Control Arm (3421)Control Arm (3421)

© 2003 By Default!


Slide 41

OPUS (TIMI-16)OPUS (TIMI-16)30 Day Mortality30 Day Mortality

Control Intervention

Period Events Patients Event Rate (%) Events Patients Event Rate (%) p

1 0 0 0.0 0 5 0.0 *

2 0 17 0.0 0 30 0.0 *

3 0 48 0.0 1 95 1.1 1.000

4 0 135 0.0 7 249 2.8 0.102

5 1 268 0.4 11 529 2.1 0.070

6 4 463 0.9 18 951 1.9 0.173

7 5 764 0.7 33 1,528 2.2 0.008

8 8 1,089 0.7 46 2,161 2.1 0.003

9 11 1,412 0.8 54 2,815 1.9 0.004

10 16 1,805 0.9 78 3,610 2.2 <0.001

11 21 2,173 1.0 92 4,410 2.1 <0.001

12 33 2,701 1.2 109 5,447 2.0 0.011

13 46 3,360 1.4 134 6,756 2.0 0.031

14 46 3,421 1.3 134 6,867 2.0 0.031

© 2003 By Default!


Slide 42

OPUS (TIMI-16) OPUS (TIMI-16) Alert SummaryAlert Summary

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

Month

Prospective

SPC

LR Stratified SPC

BUS

© 2003 By Default!


Slide 43

CLARITY (TIMI-28) CLARITY (TIMI-28)

Setting:Setting:– 313 Hospitals in 23 Countries313 Hospitals in 23 Countries

Intervention:Intervention:– Oral Anti-Platelet Agent vs PlaceboOral Anti-Platelet Agent vs Placebo

PopulationPopulation– Intervention Arm (1751)Intervention Arm (1751)

Outcome:Outcome:– Major BleedingMajor Bleeding– DSMB concerned, but trial did not stop earlyDSMB concerned, but trial did not stop early

BaselineBaseline– Control Arm (1739)Control Arm (1739)

© 2003 By Default!


Slide 44

CLARITY (TIMI-28)CLARITY (TIMI-28)Major BleedingMajor Bleeding



1 0 4 0.0 0 1 0.0 *

2 0 13 0.0 0 12 0.0 *

3 1 27 3.7 0 23 0.0 1.000

4 2 40 5.0 1 49 2.0 0.586

5 2 73 2.7 2 83 2.4 1.000

6 3 116 2.6 5 126 4.0 0.724

7 4 168 2.4 7 173 4.0 0.548

8 4 214 1.9 9 226 4.0 0.262

9 7 276 2.5 10 284 3.5 0.624

10 8 337 2.4 12 361 3.3 0.502

11 10 424 2.4 15 450 3.3 0.423

© 2003 By Default!


Slide 45

CLARITY (TIMI-28)CLARITY (TIMI-28)Major BleedingMajor Bleeding



12 11 536 2.1 16 554 2.9 0.438

13 13 639 2.0 18 649 2.8 0.468

14 17 776 2.2 22 780 2.8 0.517

15 17 892 1.9 23 926 2.5 0.427

16 18 1,041 1.7 28 1,058 2.6 0.180

17 20 1,192 1.7 31 1,195 2.6 0.156

18 24 1,314 1.8 31 1,327 2.3 0.414

19 25 1,457 1.7 31 1,459 2.1 0.500

20 26 1,584 1.6 32 1,606 2.0 0.509

21 30 1,739 1.7 34 1,751 1.9 0.706

© 2003 By Default!


Slide 46

CLARITY (TIMI-28)CLARITY (TIMI-28)Alert SummaryAlert Summary

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Month

Prospective

SPC

LR Stratified SPC

BUS

© 2003 By Default!


Slide 47

OPUS /CLARITYOPUS /CLARITYConclusionsConclusions

SPC performed well in the positive study, but SPC performed well in the positive study, but did have some false positive alerts in the did have some false positive alerts in the negative studynegative study

LR stratified SPC failed to alert early in the LR stratified SPC failed to alert early in the positive study, but performed well in the positive study, but performed well in the negative studynegative study

BUS was more specific than SPC in both BUS was more specific than SPC in both studiesstudies

© 2003 By Default!


Slide 48

Sensitivity AnalysisSensitivity Analysis


Population:Population:– All patients undergoing percutaneous coronary All patients undergoing percutaneous coronary

intervention (6175)intervention (6175)

Outcome:Outcome:– Post-procedural major adverse cardiac events (403)Post-procedural major adverse cardiac events (403)

• DeathDeath• Post-Procedural Myocardial InfarctionPost-Procedural Myocardial Infarction• Repeat VascularizationRepeat Vascularization

Baseline:Baseline:– Arbitrarily set event rates and sample sizesArbitrarily set event rates and sample sizes

© 2003 By Default!


Slide 49

Sensitivity AnalysisSensitivity AnalysisResultsResults

6158

5552

5047

4441

42 4

2 92 92

19

3

19

3

40

5

40

5

85

1

85

1

17

87

17

87

0

5

10

15

20

# Alerts

Observed Rate (% )

© 2003 By Default!


Slide 50

Clinical AlertClinical Alert


Population:Population:– All patients receiving a vascular closure device All patients receiving a vascular closure device

after percutaneous coronary intervention (3947)after percutaneous coronary intervention (3947)

Outcome:Outcome:– Retroperitoneal Hemorrhage (25)Retroperitoneal Hemorrhage (25)

Baseline:Baseline:– Stanford University Data (2000 – 2004)Stanford University Data (2000 – 2004)

© 2003 By Default!


Slide 52

Manual ReviewManual Review

Triggered root cause analysisTriggered root cause analysis

Manual chart review and multivariable Manual chart review and multivariable analsysisanalsysis

Final Result: Not related, confounded by Final Result: Not related, confounded by indicationindication

© 2003 By Default!


Slide 53

Future WorkFuture WorkMethodologyMethodology

Address Calibration ConcernsAddress Calibration Concerns

– Recalibration of Logistic Regression modelsRecalibration of Logistic Regression models

– Development of Machine Learning Risk Development of Machine Learning Risk Prediction ModelsPrediction Models

Address BUS InsensitivityAddress BUS Insensitivity

– Incorporate data weight decay over timeIncorporate data weight decay over time

© 2003 By Default!


Slide 54

Future Work Future Work ApplicationApplication

Outcomes DB

DELTAAgent

Brigham and Women’s Hospital

DELTA reports

DELTA Server

Massachusetts Data Analysis Center (Mass-DAC)

SMTP E-mail Server

E-mail alert

Outcomes DB

DELTAAgent

Massachusetts General Hospital

Outcomes DB

DELTAAgent

St. Elizabeth’s Medical Center

MA Claims DB

MA Death Index

© 2003 By Default!


Slide 55

Future WorkFuture Work

New Medication Outcomes SurveillanceNew Medication Outcomes Surveillance

Inpatient (versus Outpatient)Inpatient (versus Outpatient)

– More frequent monitoringMore frequent monitoring

– Higher quality source dataHigher quality source data

– Outcomes easier to captureOutcomes easier to capture

© 2003 By Default!


Slide 56

Future WorkFuture WorkDevelop Data RepositoryDevelop Data Repository

PatientDemographics

Local Institution

DELTA Server

CentralizedData

Repository

ComputerizedOrder Entry

Laboratory

Progress Notes(NLP)

HospitalBilling

MedicationAdministration

RadiologyIS

State DeathIndex

OtherOutcomes

© 2003 By Default!


Slide 57

Future WorkFuture WorkInitial FrameworkInitial Framework

New medication laboratory monitoring New medication laboratory monitoring protocolprotocol

Standard measures that are most commonly Standard measures that are most commonly affected in new medicationsaffected in new medications– AST, ALT, Creatinine, WBC, PlateletsAST, ALT, Creatinine, WBC, Platelets

Establish reasonable baselinesEstablish reasonable baselines– Closely Related medication lab resultsClosely Related medication lab results– Unrelated medication lab resultsUnrelated medication lab results– Expert Panel EstimationExpert Panel Estimation

© 2003 By Default!


Slide 58

AcknowledgementsAcknowledgements

MentorsMentors– Lucila Ohno-Machado, MD, PhDLucila Ohno-Machado, MD, PhD– Frederic S. Resnic, MD, MSFrederic S. Resnic, MD, MS

CollaboratorsCollaborators– Nipun Arora, MDNipun Arora, MD– Sharon Lise-Normand, PhDSharon Lise-Normand, PhD– Ewout Steyerberg, PhDEwout Steyerberg, PhD

Programming TeamProgramming Team– Richard CopeRichard Cope– Barry CoflanBarry Coflan– Atul TatkeAtul Tatke

FundingFunding– NLM R01-LM-08142NLM R01-LM-08142– NLM T15-LM-07092NLM T15-LM-07092

© 2003 By Default!


Slide 59

Michael Matheny, MD MS Michael Matheny, MD MS

[email protected]@dsg.harvard.edu Brigham & Women’s HospitalBrigham & Women’s Hospital

Thorn 309Thorn 30975 Francis Street75 Francis Street

Boston, MA 02115Boston, MA 02115

The EndThe End

© 2003 By Default! A Free sample background from Slide 1 Methodologies and Automated Applications...

Documents

Transcript of © 2003 By Default! A Free sample background from Slide 1 Methodologies and Automated Applications...