The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media &...
Transcript of The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media &...
![Page 1: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/1.jpg)
The Advanced Reality Lab
Understanding human-machine confluence in the 21st century
Brain-computer interface, virtual reality, media & brain
![Page 2: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/2.jpg)
ISVAR, April 2016
Brain-Computer Interface (BCI)
BCI
Application
closed
loop
system
feedback
brain signal control signal
![Page 3: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/3.jpg)
BCI: Signals
ISVAR, April 2016
EEG NIRS
fMRI MEG
Electrical activity Blood oxygen level
![Page 4: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/4.jpg)
BCI: Signals
ISVAR, April 2016
EEG NIRS
fMRI MEG
Electrical activity Blood oxygen level
expensive
![Page 5: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/5.jpg)
BCI: Signals
ISVAR, April 2016
EEG NIRS
fMRI MEG
Electrical activity Blood oxygen level
Low spatial resolution
![Page 6: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/6.jpg)
BCI: Signals
ISVAR, April 2016
EEG NIRS
fMRI MEG
Electrical activity Blood oxygen level
Low temporal resolution
![Page 7: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/7.jpg)
BCI: Invasive
ISVAR, April 2016
ECoG Implanted electrode array (Hochberg et al, Nature, 2012)
![Page 8: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/8.jpg)
BCI: Consumer level devices
ISVAR, April 2016
![Page 9: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/9.jpg)
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
![Page 10: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/10.jpg)
BCI based on EEG and motor imagery
Pfurtscheller and Neuper, Proceedings IEEE, 2001
ISVAR, April 2016
![Page 11: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/11.jpg)
Graz-BCI and UCL-CAVE
ISVAR, April 2016
Pfurtscheller et al., Brain
Research., 2006,
Friedman et al., HCI J 2010,
Leeb et al. Presence, 2007
![Page 12: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/12.jpg)
BCI-VR by a tetraplegic patient
ISVAR, April 2016
Leeb et al., Computational Intelligence and Neuroscience, 2007 Pfurtscheller et al., Int J Psychophysiology, 2007
![Page 13: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/13.jpg)
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
![Page 14: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/14.jpg)
Graz-BCI and UCL-CAVE
ISVAR, April 2016
Friedman et al., GRAPP, 2007
![Page 15: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/15.jpg)
Real-time fMRI
ISVAR, April 2016
Cohen et al., IEEE Conf Biomedical Robotics, 2012 Cohen et al., J Neural Eng, 2014
robot - Video
1Video 2Video
![Page 16: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/16.jpg)
3 classes: Left hand, right hand, and legs
ISVAR, April 2016
time
% signal change
![Page 17: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/17.jpg)
Whole brain machine learning
ISVAR, April 2016
![Page 18: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/18.jpg)
Classification Results
ISVAR, April 2016
![Page 19: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/19.jpg)
Amp - 95.8%, ctrl - 97.2 % (no sig difference)
Cue-based performance (4 classes)
![Page 20: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/20.jpg)
Cue FC
A gallery visualization of the left vs right contrast using a p<0.05 for the cue-based and free choice tasks, for all subjects from the amputees and the control group. For 'PW' the inflated brain is a mapping of the flat brain below. The red arrows represent the dominant hand and intact hand for the control and amputees groups, respectively.
![Page 21: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/21.jpg)
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
![Page 22: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/22.jpg)
Steady-state visually evoked potential (SSVEP)-based BCI
ISVAR, April 2016
![Page 23: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/23.jpg)
ISVAR, April 2016
“Unconscious” BCI
0
10
20
30
40
50
60
70
80
90
Experiment Sham
Mean of minimum error rate
Experiment Sham
Minimum error rates between groups at optimal
accuracy time stamp (11-12 seconds, 6 seconds post
stimuli onset), t (19) = 2.225, p < 0.05
![Page 24: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/24.jpg)
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
![Page 26: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/26.jpg)
Thought-controlled VR: Paradigms
– Navigation (controlling the viewpoint)
– Controlling an avatar
– Controlling the world directly
• In place control
• Hybrid control
– Beyond control
ISVAR, April 2016
![Page 27: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/27.jpg)
Neurofeedback
ISVAR, April 2016
![Page 28: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/28.jpg)
New Paradigms: Beyond Control
• Passive BCI
• Affective Computing
• Next-generation Neurofeedback
ISVAR, April 2016
![Page 29: The Advanced Reality Lab - WordPress.com · Brain-computer interface, virtual reality, media & brain . ISVAR, April 2016 Brain-Computer Interface (BCI) BCI Application closed loop](https://reader033.fdocuments.us/reader033/viewer/2022050507/5f98d7fa709d805b5658d340/html5/thumbnails/29.jpg)
Thank you
Doron Friedman
http://avl.idc.ac.il
ISVAR, April 2016