Playing Pinball with non-invasive BCI - Brain Products Videos of BBCI-controlled pinball machine,...
1
Results Michael Tangermann, M. Krauledat, K. Grzeska, M. Sagebaum, C. Vidaurre, B. Blankertz, K.-R.Müller, Berlin Institute of Technology. Email: [email protected] Common Spatial Patterns Experiment Playing Pinball with non-invasive BCI Intro Brain-Computer Interface rand none Control Mode Performance Comparison Four Subjects bbci rand none Control Mode bbci 0 10 20 30 40 50 60 Ball Duration [s] High-Quality Shots per Ball 0 1 2 3 4 5 6 7 -15 -10 -5 0 5 10 Percentage 0 10 20 30 40 Million Points per Game n=42 n=43 n=42 n=543 n=346 n=490 Difference of normalized histograms: (bci control) - (rand control) Supported by Filter (FQ / spatial) Classifier Low-level controller Amplifier / Digitizer Paddle control signal EEG Player Feedback - Control by thought in single-trial, no peripheral nerves involved - Purpose: communication for paralyzed patients, rehabilitation training, mobility, entertainment / gaming - Invasive signals (ECoG, intracortical, spike data) - Non-invasive signals (EEG, MEG, fMRI, NIRS) Characteristics of non-invasive signals: - High signal noise even in lab - High variance, non-stationarities - High dimensional data - Control slow [Bit/s] - Intrinsic delay by EEG features and BCI system (x*100ms) - Control not precise in time (response to cues) Calibration setup (individual for each subject) - 6 healthy subjects experienced in BCI in front of monitor - Recording of brain activity with 64-channel EEG - Calibration of BCI with 2 cued motor imagery tasks (20min) - Filter signals to subject specific mu-rhythm frequency band - Determine class-discriminant spatial filters with - Filter EEG data, calculate log-bandpower features - Training of regularized LDA on 75 to 100 trials/class - Offline cross-validation error < 10% Feedback setup with online BCI control (bbci) - Subjects seated in front of sightly pinball machine, hands relaxed - Asynchronous control of left and right pinball paddle by 2 motor imaginations - Preprocessing and filtering as determined in calibration - Continuous classification of EEG signals - Low level controller translates classifier output to control signal (thresholding, rebound, refractory times) - At least 10 games of 10 balls played Pseudo random control (rand) - Replay of previously recorded EEG controls paddles (not synchronized with ball position) Analysis of control modes (bbci, rand, no) - Video analysis of high-quality shots - Points per game - Playing duration per ball CSP modified Is real-time control possible with non-invasive BCI? Testbed: BBCI controls Addams Family pinball machine - Physical device, complex environment - Visual and acoustic distractors - Immersive: sounds, complex ball trajectories, emotions - Fast and precisely-timed control necessary - Interaction Control ability 3 subjects gained good control and enjoyed playing 1 subject gained limited control only, but enjoyed gaming 2 subjects could not establish reliable control (excluded) Control quality - Control mode bbci significantly better than rand - Longer gaming, more points, more high-quality shots! Research Questions Study relevant to rehabilitation since it explores the limits of BCI with respect to timing, dynamics and speed of interaction in a difficult real-time task. Study reveals, that appealing control is possible for non-invasive, low-risc, EEG-based BCI systems. Future research topics: - Dynamics of brain control - Mental state monitoring during successful and missed trials. Discussion CSP: supervised decomposition of multi-channel data for two classes. Goal: dimensionality reduction of multi-channel signal x(t) from C channels to C surrogate channels. Class-discriminant subspaces in columns (spatial 0 filters) of projection matrix W: Calculation of CSP Let be the class covariance matrices ( ). CSP analysis consists of calculation of matrix W and diagonal matrix D (with elements in [0,1] such that and This can be solved as a generalized eigenvalue problem. Retain 2-3 columns of W with highest eigenvalues and 2-3 with lowest eigenvalues to maximize variance of projected signals w.r.t. class 1 and class 2. References & Material [1] Benjamin Blankertz, Ryota Tomioka, Steven Lemm, Motoaki Kawanabe, and Klaus-Robert Müller. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Proc Magazine, 25(1):41–56, January 2008. [2] Klaus-Robert Müller, Michael Tangermann, Guido Dornhege, Matthias Krauledat, Gabriel Curio, and Benjamin Blankertz. Machine learning for real-time single-trial EEG-analysis: From brain-computer interfacing to mental state monitoring. J Neurosci Methods, 167(1):82–90, 2008. [3] L.R. Hochberg, M.D. Serruya, G.M. Friehs, J.A. Mukand, M. Saleh, A.H. Caplan, A. Branner, D. Chen, R.D. Penn, and J.P. Donoghue. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature, 442(7099):164–171, July 2006. http://www.bbci.de/supplementary Videos of BBCI-controlled pinball machine, overview lab setup, EEG recording, amplification and generation of pinball paddle control signals. Split-screen comparison of bbci control and pseudo-random control, longer gaming scenes, high-res images.
Transcript of Playing Pinball with non-invasive BCI - Brain Products Videos of BBCI-controlled pinball machine,...
Results
Michael Tangermann, M. Krauledat, K. Grzeska, M. Sagebaum, C. Vidaurre, B. Blankertz, K.-R.Müller,Berlin Institute of Technology. Email: [email protected]
Common Spatial Patterns
Experiment
Playing Pinball with non-invasive BCI
Intro Brain-Computer Interface
rand none
Control Mode
Performance Comparison Four Subjects
bbci rand none
Control Mode
bbci
0
10
20
30
40
50
60
Ba
ll D
ura
tion
[s]
High-Quality Shots per Ball
0 1 2 3 4 5 6 7
-15
-10
-5
0
5
10
Pe
rce
nta
ge
0
10
20
30
40
Mill
ion
Po
ints
pe
r G
am
e
n=42 n=43 n=42n=543 n=346n=490
Difference of normalized histograms:(bci control) - (rand control)
Supported by
Filter (FQ / spatial)
Classifier
Low-levelcontroller
Amplifier / Digitizer
Paddle control signal
EEG
Player
Feedback
- Control by thought in single-trial, no peripheral nerves involved- Purpose: communication for paralyzed patients, rehabilitation training, mobility, entertainment / gaming- Invasive signals (ECoG, intracortical, spike data)- Non-invasive signals (EEG, MEG, fMRI, NIRS)
Characteristics of non-invasive signals:- High signal noise even in lab- High variance, non-stationarities- High dimensional data- Control slow [Bit/s]- Intrinsic delay by EEG features and BCI system (x*100ms)- Control not precise in time (response to cues)
Calibration setup (individual for each subject)- 6 healthy subjects experienced in BCI in front of monitor- Recording of brain activity with 64-channel EEG- Calibration of BCI with 2 cued motor imagery tasks (20min)- Filter signals to subject specific mu-rhythm frequency band- Determine class-discriminant spatial filters with - Filter EEG data, calculate log-bandpower features- Training of regularized LDA on 75 to 100 trials/class- Offline cross-validation error < 10%
Feedback setup with online BCI control (bbci)- Subjects seated in front of sightly pinball machine, hands relaxed- Asynchronous control of left and right pinball paddle by 2 motor imaginations- Preprocessing and filtering as determined in calibration- Continuous classification of EEG signals- Low level controller translates classifier output to control signal (thresholding, rebound, refractory times)- At least 10 games of 10 balls played
Pseudo random control (rand)- Replay of previously recorded EEG controls paddles (not synchronized with ball position)
Analysis of control modes (bbci, rand, no)- Video analysis of high-quality shots - Points per game- Playing duration per ball
CSP
modified
Is real-time control possible with non-invasive BCI?
Testbed: BBCI controls Addams Family pinball machine- Physical device, complex environment- Visual and acoustic distractors- Immersive: sounds, complex ball trajectories, emotions- Fast and precisely-timed control necessary- Interaction
Control ability3 subjects gained good control and enjoyed playing1 subject gained limited control only, but enjoyed gaming2 subjects could not establish reliable control (excluded)
Control quality- Control mode bbci significantly better than rand- Longer gaming, more points, more high-quality shots!
Research Questions
Study relevant to rehabilitation since it explores the limits of BCI with respect to timing, dynamics and speed of interaction in a difficult real-time task.Study reveals, that appealing control is possible for non-invasive, low-risc, EEG-based BCI systems.
Future research topics:- Dynamics of brain control- Mental state monitoring during successful and missed trials.
Discussion
CSP: supervised decomposition of multi-channel data for two classes.
Goal: dimensionality reduction of multi-channel signal x(t) from C channels toC surrogate channels. Class-discriminant subspaces in columns (spatial 0filters) of projection matrix W:
Calculation of CSPLet be the class covariance matrices ( ).CSP analysis consists of calculation of matrix W and diagonal matrix D (withelements in [0,1] such that
and
This can be solved as a generalized eigenvalue problem. Retain 2-3 columnsof W with highest eigenvalues and 2-3 with lowest eigenvalues to maximizevariance of projected signals w.r.t. class 1 and class 2.
References & Material[1] Benjamin Blankertz, Ryota Tomioka, Steven Lemm, Motoaki Kawanabe, and Klaus-Robert Müller. Optimizingspatial filters for robust EEG single-trial analysis. IEEE Signal Proc Magazine, 25(1):41–56, January 2008.
[2] Klaus-Robert Müller, Michael Tangermann, Guido Dornhege, Matthias Krauledat, Gabriel Curio, andBenjamin Blankertz. Machine learning for real-time single-trial EEG-analysis: From brain-computerinterfacing to mental state monitoring. J Neurosci Methods, 167(1):82–90, 2008.
[3] L.R. Hochberg, M.D. Serruya, G.M. Friehs, J.A. Mukand, M. Saleh, A.H. Caplan, A. Branner, D. Chen,R.D. Penn, and J.P. Donoghue. Neuronal ensemble control of prosthetic devices by a human with tetraplegia.Nature, 442(7099):164–171, July 2006.
http://www.bbci.de/supplementaryVideos of BBCI-controlled pinball machine, overview lab setup, EEG recording, amplification and generation of pinball paddle control signals. Split-screen comparison of bbci control and pseudo-random control, longer gaming scenes, high-res images.
