Statistical Assessment of Event Predictors
description
Transcript of Statistical Assessment of Event Predictors
![Page 1: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/1.jpg)
Statistical Assessment of Event Predictors
Björn Schelter
IWSP 4, Kansas City
![Page 2: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/2.jpg)
Statistical Assessment of Event Predictors and
Probabilistic Forecasting
Björn SchelterAndreas Schulze-Bonhage, Hinnerk Feldwisch,
Michael Jachan, Jens Timmer,
Klaus Lehnertz, Ralph Andrzejak, Florian Mormann
IWSP 4, Kansas City
![Page 3: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/3.jpg)
The guidelines
• Use long-term EEG data without pre-selection
• Report results for training and testing data
• Provide both sensitivity as well as specificity (time under false warning?)
• Statistically validate your results
20.04.23 3IWSP 4, Kansas City
![Page 4: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/4.jpg)
Comparison
IWSP 4, Kansas City 420.04.23
Difference vanishes for Poisson distributed seizures.
![Page 5: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/5.jpg)
Results
• Analytical significance level– Tests statistical significance – Poisson process
• Monte-Carlo based technqiues– Can test statistical significance– Can test for various properties of a given
predictor– Powerful if designed correctly and in its
asymptotic
IWSP 4, Kansas City 520.04.23
![Page 6: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/6.jpg)
What is a true prediction?
20.04.23 6IWSP 4, Kansas City
![Page 7: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/7.jpg)
Standard approach
20.04.23 7IWSP 4, Kansas City
![Page 8: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/8.jpg)
Re-raising alarms
20.04.23 8IWSP 4, Kansas City
![Page 9: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/9.jpg)
Pros and cons
• Standard approach can be assessed statistically
• Re-raising alarms can be handled statistically
BUT
Sensitivity of the random predictor is 100%
20.04.23 9IWSP 4, Kansas City
![Page 10: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/10.jpg)
Solution (?) for re-raising alarms
• By Snyder et al.:– Limit the time under warning– Statistics suggested similar to SPC statistics
• But– Time under true warning might be extremely
long
IWSP 4, Kansas City 1020.04.23
![Page 11: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/11.jpg)
General idea
IWSP 4, Kansas City 1120.04.23
![Page 12: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/12.jpg)
Transform features into probability by logistic regression
Probabilistic Features
t-1 t t+1
IWSP 4, Kansas City
![Page 13: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/13.jpg)
Sensitivity is not an appropriate measure for performance here …
Range of Brier score:
Prediction Performance: The Brier Score
[Brier, Monthly Weather Review 78, 1950]
Question: When can an estimated Brier scorebe regarded as significant ?(i.e. prediction performanceabove chance level)
constant-zero predictor
natural predictor
0.25: indecisive 50% predictor
0: perfect predictor
... Indicator of seizure occurrence (0/1)
... Brier score
IWSP 4, Kansas City
![Page 14: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/14.jpg)
• Correction for multiple testing: 5 tests
MPC: mean phase coherence DSI: dynamical similarity index
• Significant results obtained for 3/5 patients
Results: Brier-Scores
[Andrzejak et al., Phys. Rev. E, 67, 2003]
IWSP 4, Kansas City
![Page 15: Statistical Assessment of Event Predictors](https://reader036.fdocuments.us/reader036/viewer/2022081516/56813fd1550346895daab388/html5/thumbnails/15.jpg)
The Group
Andreas Schulze-Bonhage
Armin Brandt
Caronlin Gierschner
Jens Timmer
Hinnerk Feldwisch
Michael Jachan
Raimar Sandner
20.04.23 15IWSP 4, Kansas City