EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and...
Transcript of EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and...
![Page 1: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/1.jpg)
DIPARTIMENTO DI INGEGNERIA DELL’INFORMAZIONE E SCIENZE MATEMATICHE
Niccolò Meucci
a.a. 2017/2018
EEG Data Analysis
and Motion
Recognition
![Page 2: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/2.jpg)
Introduction
Where
➢ Liquidweb is a startup company operating within the Information and
Communication Technologies (ICT) field, with special focus in
Ubiquitous Computing, Mobile Services and Cloud computing
technologies.
➢ BrainControl:
a framework
based on Artificial
Intelligence for
human-machine
interaction
through bio-
feedback.
![Page 3: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/3.jpg)
Introduction
What is BrainControl?
![Page 4: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/4.jpg)
Introduction
BC: how it looks like (1/3)
![Page 5: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/5.jpg)
Introduction
BC: how it looks like (2/3)
![Page 6: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/6.jpg)
Introduction
BC: how it looks like (3/3)
![Page 7: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/7.jpg)
Introduction
BC: why do we need it
➢ ALS (Amyotrophic Lateral Sclerosis)
➢ Locked In Syndrome
![Page 8: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/8.jpg)
BC: a multi-sensory platform
Introduction
➢ Augmentative and Alternative Communication (AAC)
![Page 9: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/9.jpg)
Hand Gesture
Recognition
Accelerometer-based
gesture recognition
model
![Page 10: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/10.jpg)
Hand Gesture Recognition
Introduction
➢ Current use: the accelerometer is connected via Bluetooth to the platform.
Once “MotionSensor” interaction is selected by the user, then a threshold-
based algorithm (very sophisticated ! ) is able to discriminate among 3 different
gestures:
➢ gesture UP;
➢ gesture RIGHT;
➢ gesture LEFT.
➢ Research Question: Create a model, Machine Learning- based, that
is able to classify such gestures.
![Page 11: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/11.jpg)
Hand Gesture Recognition
Structure of research
Record data
Segmentation
Data smoothing
Dataset Creation
Feat. Extraction
Classification
Input user
Gesture
Recognition
![Page 12: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/12.jpg)
Hand Gesture Recognition
Recording data (Up, Right, Left);
➢ 3 files .txt, containing only gestures belonging to one class (Up, Right, Left);
➢ file format:
➢Reference frame:
![Page 13: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/13.jpg)
Hand Gesture Recognition
Segmentation
➢ Total number of gestures: 275
➢ 80 Up;
➢ 105 Right;
➢ 90 Left;
![Page 14: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/14.jpg)
Hand Gesture Recognition
Applying Gaussian Filter (Pre-processing)
![Page 15: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/15.jpg)
Hand Gesture Recognition
Dataset Creation
➢ number of examples: 80 per class;
➢ example dimensions: [ 20 x 3 ];
➢ dataset dimensions: [ 240 x 20 x 3 ];
➢ target each record with a label:
➢ class 1: Up;
➢ class 2: Right;
➢ class 3: Left.
Dataset label
1: [ 20 x 3 ] 1 = GestUp
… …
81: [ 20 x 3 ] 2 = GestRight
… …
161: [ 20 x 3 ] 3 = GestLeft
… …
![Page 16: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/16.jpg)
Hand Gesture Recognition
Feature Extraction
➢ for each gesture:
➢ for each axis:
➢ compute (Area, Max, Min)
➢ 9 features, for each record
Dataset
dimension
FeatureSet
dimension
240 x 60 240 x 9
![Page 17: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/17.jpg)
Hand Gesture Recognition
Classifier: Suppport Vector Machine
➢ (Multi-class) Classification problem: 3 Classes
➢ Model: Support Vector Machine
➢ It constructs a hyper-plane or set of hyper-planes, which can be used for
classification, regression, ..
➢ Intuitively, a good separation is achieved by the hyper-plane that has the
largest distance to the nearest training data points of any class (margin).
x2
x1
x2
x1
![Page 18: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/18.jpg)
Hand Gesture Recognition
Train a Classifier (Supervised Learning)
➢ Split the Feature Set in Training and Test Set;
➢ Training Set dimensions: [ 160 x 9 ] 67% of data (2/3)
➢ Test Set dimensions: [ 80 x 9 ] 33% of data (1/3)
SVM
+ Class Label
OK
NO
SVM Prediction vs Class
Label
➢ Training
➢ Test
![Page 19: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/19.jpg)
Hand Gesture Recognition
Results
Training accuracy
Test accuracy
![Page 20: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/20.jpg)
Hand Gesture Recognition
Conclusions
➢ Off-line classification of gestures (Up, Right, Left)
➢ Good set of features
➢ Good generalization capability
Challenges and further works
➢ Different features (and feature set?) ?
➢ Data augmentation
➢ Real time classification (what about segmentation??)
➢ Build up an interface (GUI) able to classify gestures in real time better than
threshold-based algorithm ( classification or verification?? )
![Page 21: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/21.jpg)
BCI and EEG
Analysis
Resting vs Push
Motor Imagery
Classification
![Page 22: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/22.jpg)
Brain Computer Interface
definition
➢ A system which allows someone to communicate information about
their mental state without the use of the peripheral nervous system.
➢ signal must come directly from the brain
![Page 23: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/23.jpg)
BCI
How does it work?
➢ BCI cycle:
12
3 4
![Page 24: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/24.jpg)
BCI
The cycle: Generation (induced)
➢ Thinking of (for instance) a left-hand movement, or actually performing it,
leads you to the same (time-frequency) patterns
➢ Motor imagery: mental process by which an individual rehearses or
simulates a given action in his/her mind without actually performing the
movement (Dickstein and Deutsch, 2007; Ge et al., 2014; Sharma et al.,
2006)
1
![Page 25: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/25.jpg)
BCI
The cycle: Measurement
2
![Page 26: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/26.jpg)
BCI
The cycle: Decoding
Ch
t
Resting or Push?
➢ In general, we don‘t know how the users mental-task is encoded
in the measured signal
![Page 27: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/27.jpg)
BCI
The cycle: Decoding C
htY
e
s
o
r
n
o
?
User A User B User C
Session 1
Session 2
Session 3
➢ Inter-subject and inter-session variability
![Page 28: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/28.jpg)
BCI
The cycle: Output
![Page 29: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/29.jpg)
BCI
BrainControl output
![Page 30: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/30.jpg)
EEG Analysis
Research Question
➢ Understand what are the differences between resting and push, and how
to discriminate them from EEG point of view
Ch
t
Resting/push?
![Page 31: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/31.jpg)
EEG Analysis
Hardware
➢ Epoc vs OpenBCI
➢ OpenBCI:
➢ 8 channels
➢ fs = 250 Hz
➢ data raw (unfiltered)
![Page 32: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/32.jpg)
EEG Analysis
Protocol and Paradigm for Data Recording
➢ Paradigm:
rp
rp
rp
rp
rp
r = Resting
p = Push
Set up Set down
t (sec)4
5 2
➢ Resting = relaxation
➢ Push = ‘ thinking about moving something ’
![Page 33: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/33.jpg)
EEG Analysis
Resting vs Push
r
p
![Page 34: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/34.jpg)
EEG Analysis
Patterns in brain signal
➢ Source signals:
![Page 35: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/35.jpg)
EEG Analysis
Patterns in brain signal
Delta δ
Theta θ
Beta β
Alpha α
noise
EEG approximation:
![Page 36: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/36.jpg)
EEG Analysis
Evidence
➢ While performing a motor imagery thought (PUSH), the power in the alpha (or
beta) band decreases (Event-related desynchronization (ERD)), over sensory
motor cortex
➢ It gets back to «normal» when the user starts thinking of relaxing (RESTING)
(Event-related synchronization (ERS))
➢ Pattern evidence:
➢ Where: sensory motor cortex.
➢ When: during motor imagery
➢ What: decreasing in power over alpha/beta bands
➢ How: pre-processing, feature extraction
Pfurtscheller G, Neuper C, Mohl W.
![Page 37: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/37.jpg)
EEG Analysis
Pipeline
Record data
Pre-processing
Filtering
Channel Selection
Power Spetrum
Plotting
Input user EEG
Resting VS
Push
![Page 38: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/38.jpg)
EEG Analysis
Record data: Notch Filter + Band Pass (1-30 Hz)
![Page 39: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/39.jpg)
EEG Analysis
Channel Selection
➢ Only sensors from somatosensory motor cortex are needed
T3
C3
C4
T4
![Page 40: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/40.jpg)
EEG Analysis
Results and Conclusions
➢ My results
➢ Conclusions:
Resting and Push can be discriminated by computing the power spectrum of
alpha and beta bands, according to the literature
➢ Literature
➢ Power spectrum
A.Sivakami and
S.Shenbaga Devi
![Page 41: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/41.jpg)
EEG Analysis
Application
Resting
Push
![Page 42: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/42.jpg)
EEG Analysis
Further works
➢ Gather more data for Statistical analysis
➢ Change features ( Spatial Filtering, …)
➢ Gather more (and more) data to train a Classifier (SVM)
➢ Eventually Real-time classification with Machine Learning Models ( BCI )
![Page 43: EEG Data Analysis and Motion Recognitionmonica/bioinfon/Slides_Meucci.pdf · EEG Data Analysis and Motion Recognition. Introduction ... Brain Computer Interface definition A system](https://reader033.fdocuments.us/reader033/viewer/2022060303/5f08cc507e708231d423c5d1/html5/thumbnails/43.jpg)
Innovative Human
Machine Interfaces