SYNESTHESI A Multimodal Communication 2008 Ingvar Lind sight taste smell hearing touch.
-
date post
21-Dec-2015 -
Category
Documents
-
view
217 -
download
2
Transcript of SYNESTHESI A Multimodal Communication 2008 Ingvar Lind sight taste smell hearing touch.
SYNESTHESI AMultimodal Communication 2008
Ingvar Lind
sight
taste
smell
hearing
touch
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Muldimodality in communication• Different modalities are used simultaneously in
multimodal communication.• The modalities interact in certain ways.• There are production and reception modalities.• The receiving modalities are received through the
senses, which translate physical properties in the world to mental modules.
• The mental modules are both experiences and unconcious simulation processes that together combine to the world we know and feel.
• The modalities are combined in an indirect fashion in multimodal communication. In synesthesia, the combination of modalities is direct and unchanging.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
What is Synesthesia?
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:• A condition in which stimulation of one sense also
evokes another sensory pathway in the brain.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:• A condition in which stimulation of one sense also
evokes another sensory pathway in the brain.• It is involuntary.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:• A condition in which stimulation of one sense also
evokes another sensory pathway in the brain.• It is involuntary.• It is automatic (consistent and permanent).
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:• A condition in which stimulation of one sense also
evokes another sensory pathway in the brain.• It is involuntary.• It is automatic (consistent and permanent).• It can be cross-modal and inter-modal (=cross-sub-
modal).
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:• A condition in which stimulation of one sense also
evokes another sensory pathway in the brain.• It is involuntary.• It is automatic (consistent and permanent).• It can be cross-modal and inter-modal (=cross-sub-
modal).• It is highly memorable.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:• A condition in which stimulation of one sense also
evokes another sensory pathway in the brain.• It is involuntary.• It is automatic (consistent and permanent).• It can be cross-modal and inter-modal (=cross-sub-
modal).• It is highly memorable.• It is paired with emotion/mood.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:• A condition in which stimulation of one sense also
evokes another sensory pathway in the brain.• It is involuntary.• It is automatic (consistent and permanent).• It can be cross-modal and inter-modal (=cross-sub-
modal).• It is highly memorable.• It is paired with emotion/mood.• It is most often unique to one person, but similarities
exist between the synesthetic types.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:• A condition in which stimulation of one sense also
evokes another sensory pathway in the brain.• It is involuntary.• It is automatic (consistent and permanent).• It can be cross-modal and inter-modal (=cross-sub-
modal).• It is highly memorable.• It is paired with emotion/mood.• It is most often unique to one person, but similarities
exist between the synesthetic types.• The synesthetic phenomena is an extension of
perception rather than a limitation, as in color blindness.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Definitions of Synesthesia:• A condition in which stimulation of one sense also
evokes another sensory pathway in the brain.• It is involuntary.• It is automatic (consistent and permanent).• It can be cross-modal and inter-modal (=cross-sub-
modal).• It is highly memorable.• It is paired with emotion/mood.• It is most often unique to one person, but similarities
exist between the synesthetic types.• The synesthetic phenomena is an extension of
perception rather than a limitation, as in color blindness.• It is directional, from one modality to another.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Examples of Synesthesia:• numbers -> shapes/colors/textures/movements
Extraordinary people - Daniel TammetPart 2: Beginning 3 min.
http://www.youtube.com/watch?v=ydiBnm1ejhM&NR
Part 3: Beginning 3 min.
http://www.youtube.com/watch?v=4BxhYPKXXnQ
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Examples of Synesthesia:• graphemes (letters) -> sounds
Sounds of the Alphabet, 1 min.
http://www.youtube.com/watch?v=XOuU0ppCcn8
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Examples of Synesthesia:• drugs -> synesthesia
Synesthesia caused by drugs, 2 min.
http://www.youtube.com/watch?v=XZ7jqgEYSLU
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Examples of Synesthesia:• music -> colors• graphemes (numbers and letters) -> colors• taste -> shape• speech sound -> taste• time units -> positions in space
Seeing Life in Colors: Crosswired Senses, ABC NEWS 3 min.http://www.youtube.com/watch?v=KApieSGlyBk
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Examples of simulated Synesthesia:• sound -> moving shapes• music -> moving lines
Synthesizer Synesthesia, 1.22 min.
http://www.youtube.com/watch?v=l4_T5VYMHqc
Toccata and Fugue in D minor, 8.34 min.
http://www.youtube.com/watch?v=ipzR9bhei_o
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Colors
Tastes
Smells
Touch
Temperature
Vibration, skin
Shape in 3D
Movements
Pain
Positions
Synesthesia Modalities and Modules
Concepts (language)
Concepts (mathematics)
Emotions
Music
Sound
Time units
Graphemes
Phonemes
Faces
Bodies
etc.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Categorical perception and Synesthesia
Categorical perception is perception of a distinct number of kinds, opposed to a perception of continuous dimensions in experience. The hue circle is an example of a continuous dimension that perception labels with a small number of hue categories: red, orange, yellow, green, blue, purple.
Categorical perception is a necessare function of perception to scale down the complexity of information from the world into a relatively small number of labels.
Synesthesia are often connections between categories of modalities rather than between continuous dimensions. This suggests that language, or other more abstract brain functions has a major role in synesthesia.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)?
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)?
1. Cross-sensory metaphores (blue music, blue feelings)
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)?
1. Cross-sensory metaphores (blue music, blue feelings)
2. Geometrical properties are common between the modalities (2D structures, 3D-structures (places, movements, textures and directions in a room in vision, auditory and tactile modalities)
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)?
1. Cross-sensory metaphores (blue music, blue feelings)
2. Geometrical properties are common between the modalities (2D structures, 3D-structures (places, movements, textures and directions in a room in vision, auditory and tactile modalities)
3. The Bouba-Kiki effect
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
Can structures/patterns of different modalities be described and compared in a functional way (similarities, differences of neural patterns underlying functional structures)?
1. Cross-sensory metaphores (blue music, blue feelings)
2. Geometrical properties are common between the modalities (2D structures, 3D-structures (places, movements, textures and directions in a room in vision, auditory and tactile modalities)
3. The Bouba-Kiki effect (unknown words suggest certain shapes)
4. Temperature and color (blue feels cold, red feels warm)
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
The mind-body-dualism
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
The mind-body-dualism
1. The mind studied with introspection, interviews and psychophysics. Only the mental functions are considered.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
The mind-body-dualism
1. The mind studied with introspection, interviews and psychophysics. Only the mental functions are considered.
2. The body as controlled by a computerlike brain, with neural network models explaining what goes on between input and output.
Can the two perspectives be fused into one? Can synesthesia show us how the mind works?
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
Theories of synesthesia
1. Module theory
Cognitive psychology: modules have separate functions in the brain. Some sensory modules are linked in synesthesia.
Synesthesia - Multimodal Communication 2008 Ingvar Lind
Synesthesia Questions
Theories of synesthesia
1. Module theory
Cognitive psychology: modules have separate functions in the brain. Some sensory modules are linked in synesthesia.
2. Matching theory
Cross-modal matching, one modality can process information patterns that another modality process: size, shape, intensity, texture, quality, time, space.
”Synesthesia”-model for a connection between color and sound
Vertical direction: Brightness
Central vertical arrow: neutral colors (black, gray, white, shiny)
Horizontal circle: Hues (red, orange, yellow, green, blue, purple, red)
Vertical direction: mental Volume of sound (or maybe Pitch)
Central vertical arrow: neutral vowels (central in the formant space)
Horizontal circle: Vowel sounds (timbre) (a, e, i, y, u, o, a)
1-1 mapping
Dotted line: strength of Hue or strength of Vowel
Synesthesia - Multimodal Communication 2008 Ingvar Lind
An experiment in Synesthesia
Make a synesthetic one-to-one connection model between the 2D vowel space and the 2D hue space, using the computer programs Praat and ColorSpace. What is the most natural connection between the two? Explain why.
Vowel space:Praat (Macintosh, Pc, Linux, SGI)http://www.fon.hum.uva.nl/praat/Praat objects – New – Sound – Create Sound from Voweleditior
Hue space:ColorSpace (Windows program, similar to Macintosh Color Picker) http://www.codeguru.com/cpp/w-d/dislog/commondialogs/article.php/c1861View – Color Space (Color Picker)
Synesthesia - Multimodal Communication 2008 Ingvar Lind
References
Sean Day, “Trends in colored letter synesthesia”, research on colored letters:http://web.mit.edu/synesthesia/www/trends.htmlSensequence, a huge webpage on synesthesia:http://www.sensequence.de/indexen.html"Art and Synesthesia: in search of the synesthetic experience":http://www.doctorhugo.org/synesthesia/art/index.html” The Phenomenology of Synaesthesia”, V.S. Ramachandran and E.M. Hubbard:http://www.imprint.co.uk/pdf/R_H-follow-up.pdf“The perceptual reality of synesthetic colors”:http://www.psy.vanderbilt.edu/faculty/blake/PDFs/PalmeriEtAl_PNAS_2002.pdfCognitive, Affective, & Behavioral Neuroscience 2004, 4 (3), 335-34,” Not all synesthetes
are created equal: Projector versus associator synesthetes”http://watarts.uwaterloo.ca/~src/papers/Dixon.CABN.2004.pdfConsciousness and Cognition 16 (2007) 913–931“Varieties of grapheme-colour synesthesia: A new theory of phenomenological and
behavioural differences”http://home.comcast.net/~sean.day/Ward%20et%20al%202007.pdfThe neuropsychology of synesthesia:http://scienceblogs.com/neurophilosophy/2007/09/the_neuropsychology_of_synest.php
Synesthesia - Multimodal Communication 2008 Ingvar Lind
References, books
Richard E. Cytowic
Synesthesia, A Union of the Senses, 1989
Richard E. Cytowic
The Man Who Tasted Shapes, 1993
Monica Vester
En värld av nyanser, 2004