Statistical Assessment of Event Predictors

15
Statistical Assessment of Event Predictors Björn Schelter IWSP 4, Kansas City

description

Statistical Assessment of Event Predictors. Björn Schelter. Statistical Assessment of Event Predictors and Probabilistic Forecasting. Björn Schelter Andreas Schulze-Bonhage, Hinnerk Feldwisch, Michael Jachan, Jens Timmer, Klaus Lehnertz, Ralph Andrzejak, Florian Mormann. The guidelines. - PowerPoint PPT Presentation

Transcript of Statistical Assessment of Event Predictors

Page 1: Statistical Assessment of  Event Predictors

Statistical Assessment of Event Predictors

Björn Schelter

IWSP 4, Kansas City

Page 2: Statistical Assessment of  Event Predictors

Statistical Assessment of Event Predictors and

Probabilistic Forecasting

Björn SchelterAndreas Schulze-Bonhage, Hinnerk Feldwisch,

Michael Jachan, Jens Timmer,

Klaus Lehnertz, Ralph Andrzejak, Florian Mormann

IWSP 4, Kansas City

Page 3: Statistical Assessment of  Event Predictors

The guidelines

• Use long-term EEG data without pre-selection

• Report results for training and testing data

• Provide both sensitivity as well as specificity (time under false warning?)

• Statistically validate your results

20.04.23 3IWSP 4, Kansas City

Page 4: Statistical Assessment of  Event Predictors

Comparison

IWSP 4, Kansas City 420.04.23

Difference vanishes for Poisson distributed seizures.

Page 5: Statistical Assessment of  Event Predictors

Results

• Analytical significance level– Tests statistical significance – Poisson process

• Monte-Carlo based technqiues– Can test statistical significance– Can test for various properties of a given

predictor– Powerful if designed correctly and in its

asymptotic

IWSP 4, Kansas City 520.04.23

Page 6: Statistical Assessment of  Event Predictors

What is a true prediction?

20.04.23 6IWSP 4, Kansas City

Page 7: Statistical Assessment of  Event Predictors

Standard approach

20.04.23 7IWSP 4, Kansas City

Page 8: Statistical Assessment of  Event Predictors

Re-raising alarms

20.04.23 8IWSP 4, Kansas City

Page 9: Statistical Assessment of  Event Predictors

Pros and cons

• Standard approach can be assessed statistically

• Re-raising alarms can be handled statistically

BUT

Sensitivity of the random predictor is 100%

20.04.23 9IWSP 4, Kansas City

Page 10: Statistical Assessment of  Event Predictors

Solution (?) for re-raising alarms

• By Snyder et al.:– Limit the time under warning– Statistics suggested similar to SPC statistics

• But– Time under true warning might be extremely

long

IWSP 4, Kansas City 1020.04.23

Page 11: Statistical Assessment of  Event Predictors

General idea

IWSP 4, Kansas City 1120.04.23

Page 12: Statistical Assessment of  Event Predictors

Transform features into probability by logistic regression

Probabilistic Features

t-1 t t+1

IWSP 4, Kansas City

Page 13: Statistical Assessment of  Event Predictors

Sensitivity is not an appropriate measure for performance here …

Range of Brier score:

Prediction Performance: The Brier Score

[Brier, Monthly Weather Review 78, 1950]

Question: When can an estimated Brier scorebe regarded as significant ?(i.e. prediction performanceabove chance level)

constant-zero predictor

natural predictor

0.25: indecisive 50% predictor

0: perfect predictor

... Indicator of seizure occurrence (0/1)

... Brier score

IWSP 4, Kansas City

Page 14: Statistical Assessment of  Event Predictors

• Correction for multiple testing: 5 tests

MPC: mean phase coherence DSI: dynamical similarity index

• Significant results obtained for 3/5 patients

Results: Brier-Scores

[Andrzejak et al., Phys. Rev. E, 67, 2003]

IWSP 4, Kansas City

Page 15: Statistical Assessment of  Event Predictors

The Group

Andreas Schulze-Bonhage

Armin Brandt

Caronlin Gierschner

Jens Timmer

Hinnerk Feldwisch

Michael Jachan

Raimar Sandner

20.04.23 15IWSP 4, Kansas City