!"#$%&'()*+',-.*-/0-1'(23)' 3'

New applications based on implicit, physiology based interaction!!

Dr. Aleksander Väljamäe Marie Sk!odowska-Curie Fellow

Decision, Emotion and Perception Lab Department of Behavioural Sciences and Learning

Linköping University, Linköping, Sweden &

Laboratory of psychophysiology Department of Higher Nervous Activity and Psychophysiology

SPbSU, Saint Petersburg, Russia &

NeuroGenetics Media Lab, CIANT, Prague, Czech Republic

!"#$%&'()*+',-.*-/0-1'(23)' ('

!"#$%&'()*+',-.*-/0-1'(23)' 4'

Born in Tallinn, USSR, in 1978 1996-2000 First degree (B.Sc.) – Telecommunication, Tallinn TU, Tallinn, Estonia 2001-2003 Second degree (M.Sc.) – Telecommunication, Chalmers University of Technology, Gothenburg, Sweden 2003-2007 Ph.D. on multisensory virtual reality – Chalmers University of Technology, Gothenburg, Sweden 2007-2010 First postdoc – neuroscience lab SPECS, University of Pompeu Fabra, Barcelona, Spain 2011-2012 Second postdoc – Graz BCI lab, TU Graz & Neuropsychology Laboratory, University of Graz, Graz, Austria

Short CV

!"#$%&'()*+',-.*-/0-1'(23)' )'

!"#$%&'()*+',-.*-/0-1'(23)' 5'

from Wolpaw et al. 2002

!"#$%&'()*+',-.*-/0-1'(23)' 6'

from Wolpaw et al. 2002

!"#$%&'()*+',-.*-/0-1'(23)' 7'

'Implicit, physiology based interaction Physiological computing provides new channel to the conventional Human Computer Interaction paradigm. My interest is implicit interaction with multisensory media and using it for health and art applications.

1. Subtopic one: Cinema and neuroscience: recent research developments and future applications

2. Subtopic two: B-Reactable: multimodal tabletop system for collaborative physiology monitoring and training

!"#$%&'()*+',-.*-/0-1'(23)' 8'

Media as a constructed... but "real" experience

!"#$%&'()*+',-.*-/0-1'(23)' 9'

MEDIA AS CONSTRUCTED EXPERIENCES

Q1: What is neurocinema and enactive cinema?

Q2: How in can be used in art and research?

!"#$%&'()*+',-.*-/0-1'(23)' 32'

neuro cinematics

!"#$%&'()*+',-.*-/0-1'(23)' 33'

Nishimoto et al., 2011, Current Biology

Work by Gallant Lab at UC Berkeley

!"#$%&'()*+',-.*-/0-1'(23)' 3('

Measuring Inter-Subject Correlation - ISC Neurocinematics –

"The ability to measure the effect of films on viewers’ brains with high spatial and temporal precisions can provide a new analytical paradigm for assessing and analyzing different aspects of films, film genres, and cinematic styles."

Hasson et al., Projections 2008

!"#$%&'()*+',-.*-/0-1'(23)' 34'

Inter-Subject Correlations for different films (1)

Bang! You're Dead (Hitchcock, 1961)

Hasson et al., Projections, 2008

!"#$%&'()*+',-.*-/0-1'(23)' 3)'

Inter-Subject Correlations for different films (2)

Hasson et al., Projections, 2008

!"#$%&'()*+',-.*-/0-1'(23)' 35'

Interactive cinema

!"#$%&'()*+',-.*-/0-1'(23)' 36'

Creating interactive narrative

!! Script-Based Interactive Narrative: - all scenes are pre-scripted - linear or non-linear pre-dermined branchining structure

!! Emergence-Based Interactive Narrative: - agents autonomy (goal-directed and reactive actions) - story is created by agents and user interaction

!! Plan-Based Interactive Narrative: - agents autonomy - narrative manager dynamically evaluates the actions of agents and user and channels them towards a coherent story

!"#$%&'()*+',-.*-/0-1'(23)' 37'

'Affective chronometry and database cinema

!"#$%&'()*+',-.*-/0-1'(23)' 38'

Alexander Ponomarev, INFO-SHOWER, Russian Pavillon, Venice Bienalle 2007

Pulling the armatures one can regulate the speed of the pictureflood. The content itself is mixed with news, entertainment, porn, musicvideos etc.

Database media and cinema

http://tagr.tv http://www.youtube.com/watch?v=NEfVjAbMsZQ

!"#$%&'()*+',-.*-/0-1'(23)' 39'

Interactive cinema

neuro cinematics

enactive cinema

Interactive cinema based on physiological sensing

!"#$%&'()*+',-.*-/0-1'(23)' (2'

Using film for studying socio-emotional interaction

Tikka et al., 2012

!"#$%&'()*+',-.*-/0-1'(23)' (3'

Implicit interaction: wearable systems

www.neurosky.com

www.emotiv.com

www.smartex.it

www.starlab.es

!"#$%&'()*+',-.*-/0-1'(23)' (('

"Room of desires" installation (Smetana, 1996) http://www.smetana.cz

!"#$%&'()*+',-.*-/0-1'(23)' (4'

"Room of desires" installation – system architecture

!"#$%&'()*+',-.*-/0-1'(23)' ()'

Enactive cinema: "Obsession" installation, 2005

Tikka et al., 2006

!"#$%&'()*+',-.*-/0-1'(23)' (5'

"Obsession" installation (Tikka, 2005)

Tikka et al., 2006

!"#$%&'()*+',-.*-/0-1'(23)' (6'

Kaipainen et al., 2011

Enactive Cinema project: ontospace narrative engine

!"#$%&'()*+',-.*-/0-1'(23)' (7'

SPECS-UPF research lines

!"#$%&'()*+',-.*-/0-1'(23)' (8'

'Affective narratives as guided experience

!"#$%&'()*+',-.*-/0-1'(23)' (9'

EDA-based modulation of video effects

!"#$%&'()*+',-.*-/0-1'(23)' 42'

'Neurofeedback in XIM – self-motion paradigm

!"#$%&'()*+',-.*-/0-1'(23)' 43'

Controlled stimulus – affective databases

HAU

LAU

HAP

LAP

!"#$%&'()*+',-.*-/0-1'(23)' 4('

Studying transitions using subsequent picture-pairs

HAU

LAP

!"#$%&'()*+',-.*-/0-1'(23)' 44'

'(A) Low arousal pleasant -> High arousal unpleasant

3 s pause between pictures 6 s pause between pictures

Patients: 22 migraineurs; age and gender matched controls: 12

Controls show positive valence supression, but patients do not

!"#$%&'()*+',-.*-/0-1'(23)' 4)'

!"#$%&'()*+',-.*-/0-1'(23)' 45'

Can enactive cinema be used in science?

Topics that can be addressed by enactive cinema: - temporal dynamics of cognition, emotion and perception - unconscious processing - decision making - emotional synchronization, collective experience - imaging genetics

!"#$%&'()*+',-.*-/0-1'(23)' 46'

Conclusions / Topics for discussions

Enactive cinema might have a big potential as a scientific tool

!"#$%&'()*+',-.*-/0-1'(23)' 47'

Conclusions / Topics for discussions

Enactive cinema might have a big potential as a scientific tool

What are the areas of common interest: - research questions (creating a demand) - research methodologies (e.g. ERPs during film presentation) - common video databases and related infrastructure (e.g. "narratives" engine) - ...

!"#$%&'()*+',-.*-/0-1'(23)' 48'

Radically new neurocinematic experiences?

Gerson et al., IEEE Trans. Neural Syst. 2006

Rapid Serial Visual Presentation – RSVP- sequence for "human visual processors"

100 ms per image

!"#$%&'()*+',-.*-/0-1'(23)' 49'

:;<='<>'?@1A'0B'$11<C'D<11EF&'(224'

!"#$%&'()*+',-.*-/0-1'(23)' )2'

Brain to Brain communication

.-1F<G@C'@H-IJK-'LEIJ<G@1B'

.-1F<G@C'@H-IJK-'LEIJ<G@1B'

!"#$%&'()*+',-.*-/0-1'(23)' )3'

New installation in progress

!"#$%&'()*+',-.*-/0-1'(23)' )('

B-Reactable project: 2013-2015

!"#$%&'()*+',-.*-/0-1'(23)' )4'

Collaborations: - Aleksei Ossadchi, Saint Petersburg State University - Prof. Juri D. Kropotov, Natalia Shemyakina, Institute of Human Brain, Russian Academy of Sciences - Prof. Daniel Västfjäll, Linköping University, Linköping, Sweden - Prof. Sergi Jorda, Sebastian Mealla, University of Pompeu Fabra, Barcelona, Spain

“B-Reactable: multimodal tabletop system for collaborative physiology monitoring and training" https://vimeo.com/14675468

B-Reactable project: 2013-2015

!"#$%&'()*+',-.*-/0-1'(23)' ))'

'First B-Reactable experiments

!"#$%&'()*+',-.*-/0-1'(23)' )5'

'Collaborative tasks using EEG monitoring

!"#$%&'()*+',-.*-/0-1'(23)' )6'

'Testing new setup at Sankt-Petersburg

!"#$%&'()*+',-.*-/0-1'(23)' )7'

'New B-Reactable experiments: system architecture

!"#$%&'()*+',-.*-/0-1'(23)' )8'

'New B-Reactable experiments: test conditions

!"#$%&'()*+',-.*-/0-1'(23)' )9'

'B-Reactable work in progress

!"#$%&'()*+',-.*-/0-1'(23)' 52'

'New B-Reactable experiments: results

!"#$%&'()*+',-.*-/0-1'(23)' 53'

Thank you for your attention!

aleksander.valjamae@gmail.com

Web: http://www.ibl.liu.se/medarbetare/valjamae-aleksander?l=en