Quantitative methods of signal detection
-
Upload
francois-maignen -
Category
Health & Medicine
-
view
296 -
download
0
Transcript of Quantitative methods of signal detection
![Page 1: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/1.jpg)
An agency of the European Union
Presented by: Name SurnamePosition/organisational descriptor
Quantitative methods of signal detectionFrançois MAIGNEN
![Page 2: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/2.jpg)
2
Learning objectives•Current methods: Disproportionality analysis•Improvement of the current methods: Bayesian methods and (Bayesian) log-linear models•Comparison of the methods (limitations, stats vs clinical)•Star wars – a new hope? Longitudinal health records: better? Different and not devoid of biases•Future directions
![Page 3: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/3.jpg)
3
Measures of disproportionate reportingMost of the methods routinely used in pharmacovigilance (spontaneous reporting systems) databases are based on measures of disproportionate reporting (i.e. ROR, PRR, BCPNN, MGPS, etc …).Basically: “Observed vs Expected” analysis in a given database i.e. % of reports involving a given reaction for a given medicine compared to the % of reports involving this reaction on the whole database
![Page 4: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/4.jpg)
4
A spontaneous reporting system database
SRS Drug 1 Drug 2 Drug 3 Drug 4 Drug 5 Drug 6 Drug 7 … Drug N
Event 1 n11 n12 n13 n14 n15 n16 n17 … n1N
Event 2 n21 n22 n23 n24 n25 n26 n27 … n2N
Event 3 n31 n32 … … … … … … n3N
Event 4 n41 n42 … … … … … … n4N
Event 5 n51 n52 … … … … … … n5N
Event 6 n61 n62 … … … … … … n6N
… … … … … … … … … …
Event P nP1 nP2 nP3 nP4 nP5 nP6 nP7 … nPN
![Page 5: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/5.jpg)
5
Drug 1 All other medicinal products
Total
Event 1 a c
All other reaction terms
b d
Total N = a + b + c + d
c + d
a + c
a + b
Proportional Reporting Ratio
![Page 6: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/6.jpg)
6
Proportional Reporting RatioPRR = a/(a+b) / c/(c+d) WHAT DOES THAT MEAN IN PRACTICAL TERMS?a/(a+b) = Proportion of reports involving a specific adverse event among all the reports involving DRUG Ac/(c+d) = Proportion of reports involving THE SAME adverse event among all the reports of your database but DRUG A
![Page 7: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/7.jpg)
7
Proportional Reporting RatioIf the rate of reporting of AE for drug 1 is similar to the rate of reporting of this AE for all the other products of the database, the PRR will be equal to 1 (same proportion of reports involving the reaction for drug A than for the other drugs) …
BUT … If the reaction is proportionately MORE reported with drug A than for the other products, the PRR will be increased (typically > 1). DIS-PROPORTIONALITY of reporting
![Page 8: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/8.jpg)
8
Disproportionality analysis (example). CNS drug for which the total No of reports is 400, of these 20 reports of diarrhoea. All other products in the database (1 million reports excluding reports involving drug A), of these 50,000 reports of diarrhoea.
PRR = [20/400] / [50,000/1,000,000] = 1 (no SDR)
![Page 9: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/9.jpg)
9
Disproportionality analysis (example)CNS drug for which the total No of reports is 400, of these 40 reports of drowsiness. All other products in the database (1 million reports excluding reports involving drug A), of these 25,000 reports of diarrhoea.
PRR = [40/400] / [25,000/1,000,000] = 4 (presence of a SDR)
![Page 10: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/10.jpg)
10
Improvements of these methods•Considering possible confounding factors: stratification and log-linear models (ROR – see work from E. Van Puijenbroek)•Trying to circumvent low expected values or low case counts: Bayesian models (A. Bate & W. DuMouchel)•Other regression methods: LASSO and Bayesian logistic regressions (N. Noren, D. Madigan)•Clinical relevance not always clear or demonstrated•Some methods can be computationally demanding
![Page 11: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/11.jpg)
11
Bayesian methodsBCPNN and MGPS rely on the same principle of conjugate prior distributions:•These methods will shrink the value of the measure of disproportionality using a Bayesian approach (prior based on existing dataset)•BCPNN: cell counts ~ Binomial dist., conjugate prior = beta•MGPS: cell counts ~ Poisson, conjugate prior = Gamma (mixture of Gammas).
![Page 12: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/12.jpg)
12
Bayesian methodsAssume binomial y=7 successes, 20 trials.Non informative prior = Beta (2,4)
![Page 13: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/13.jpg)
13
Thresholds - ARBITRARYAll these methods provide a ranking …Thresholds = arbitraryTrade-off between•Reviewing too many drug-event pairs(loss of operational benefit)•Missing some signals
No ADR ADR
![Page 14: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/14.jpg)
14
Comparison of the methodsMethodological difficultiesNo gold standard / no standardised reference methodImprecision of what constitutes a signalRetrospective vs prospective evaluationImportance of clinical judgement. The added value of clinical evaluation is currently unknown (if any).
![Page 15: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/15.jpg)
15
Comparison of methods
1
2
3
4
7
11
6
910
12
5
8
1
2
3
4
7
11
6
910
12
5
8
4
2
3
1
7
11
6
910
12
5
8
Meth.1
Meth.2
Meth.3
• Threshold 1: Meth. 2 = 5 true signals, meth. 1/3 = 4.• Threshold 1+2: Meth. 2=Meth.3• First 5 signals: Meth. 1 ≠ Meth. 2 = 3.
ADR No ADR
![Page 16: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/16.jpg)
16
Comparison of methods (PRR vs BCPNN)
![Page 17: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/17.jpg)
17
Performances of these methodsOperational benefit (screening of large databases)Anecdotal evidence (in opposition to structured) of signals discovered thanks to the quantitative methods (recent examples incl. D:A:D and MI)Time benefit in some cases (Hochberg & EV study)NND ~ 7/15 (depending whether the study is retrospective or prospective)Idea: Quant. Methods + DMEs/TMEs
![Page 18: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/18.jpg)
18
New approaches to signal detectionDeviation of Obs. vs Expect. distr. from a fitted distribution (Jim)Modelling of the hazard function of the time to onset (DSRU / François) hazard # mechanismUse of longitudinal databases (record linkage and electronic health records – OMOP / Noren / Callreus) ~ incidence rate ratio• Same patients different time windows (A. Bate)• Hospital records of different patients (T. Callreus)
![Page 19: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/19.jpg)
19
Merci Andrew
![Page 20: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/20.jpg)
20
![Page 21: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/21.jpg)
21
Interpretation / LimitationsPatients prescribed a PPI are associated withacute pancreatitis in the month after thePrescription (ICdiff positive)But graph shows that these patients are generallikely to have acute pancreatitis around the timeof PPI prescribing (In agreement with confounding by indication)
![Page 22: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/22.jpg)
22
Record linkage
![Page 23: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/23.jpg)
23
![Page 24: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/24.jpg)
24
Interpretation / limitationsBias associated with the hospitalisation (confirmed by later events which occurred remotely after the administration of the vaccine)Spurious / unexplained associationsRely heavily on temporal association (Post hoc ergo propter hoc)
![Page 25: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/25.jpg)
25
Longitudinal health recordsPowerful to detect some associationsEven if reporting artefacts do not influence this method, not devoid of other biases (selection, protopathic, misclassification, etc …)Much more complicated to implement (very large datasets, confidentiality aspects, linkage of records and interoperability of databases, etc …)
![Page 26: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/26.jpg)
26
Future directions (?)•Performances in a prospective setting, added value of clinical judgement needs to be quantified•Duplicate detection•Other approaches to signal detection (use all the information included in the reports, PK/PD properties of the substances)•Surveillance networks (incl. spontaneous reporting)
![Page 27: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/27.jpg)
27
Key elements•PRR is a measure of disproportionality of reporting in a specific database (observed vs expected value computed on the whole database)•A SDR indicates a possible signal which needs to be medically confirmed•All other methods proceed from the same logic•Data mining on EHR will probably improve the detection of signals (organised method of collection) but is not free of biases and is resources demanding
![Page 28: Quantitative methods of signal detection](https://reader035.fdocuments.us/reader035/viewer/2022070518/58e901b81a28abc7598b4dab/html5/thumbnails/28.jpg)
28
BibliographyGuideline on the use of statistical signal detection methods in EVDAS
http://www.ema.europa.eu/pdfs/human/phvwp/10646406enfin.pdf
Bate & Evans quantitaitve signal detection using ADR reporting. PDS 2009.
Hauben & Bate. Decision support methods for the detection of adverse events in post-marketing data. Drug Discovery Today. April 2009.
Evaluation of Signal Detection Methods for use in Prospective Post Licensure Medical Product Safety Surveillance [FDA Sentinel Initiative Safety Signal Identification Contract] – Report
http://www.fda.gov/Safety/FDAsSentinelInitiative/ucm149343.htm