Music Aided Design:The Foundations of Spatial Music

Felix Faire BA Arch Dissertation - Click to open pdf

Acknowledgements:

I would like to thank Prof. François Penz for his expertise, guidance and humour; Amir Soltani for his enthusiasm and support;

And all the friends who volunteered to be experimented on:

Adam, Sohanna, Matt, Max, Lucy, Song, Jazz, Lauren, Katie, Mica, Iain, Immy, Fiona, Daniel, Elly, Miranda, Charlie, Sam, Freddy, Emma, Phoebe, Sophia, Alice, Kitty, Livvy

and Francois.

2

9353 words

2 3

Abstract.

Key words: Embodied music cognition, sound, space, interaction, association

This dissertation hypothesises that our perceptions of music and space are cognitively connected through embodied interaction with the world. These perceptual connections will be initially revealed by looking at our structures of language and speech, and then examining physical and audible precedents in the work of Iannis Xenakis, Oskar Fischinger and Mark Applebaum. The hypothesis will be further developed by exploring the shared neurological processes in listening to music and the navigation of spatial environments. The theoretical framework developed from the research is tested with a series of spatio-musical interactions tested by 24 volunteers. The results are discussed in relation to their application to musical composition, architecture and urbanism, speculating the potential for a new paradigm in Music Aided Design.

Contents.

Introduction:

Precedents:

Music Signification:

The Musical Score:

Animated Music:

Sound Gestures:

The Psychogeography of Music:

Experimental Research:

Pitch and Height:

Sonic Compass:

Sounds in Space:

Spaces of Sound:

Music and Motion:

Haptic Music:

Experiment Data:

Discussion:

List of Illustrations:

Bibliography:

Appendix:

5

7

9

13

15

18

21

22

23

24

25

26

27

29

30

33

34

36

4

“To most of us music suggests definite mental images of form and colour. The

picture you are about to see is a novel scientific experiment - it’s object is to convey

these mental images in visual form.” - Oskar Fischinger 1938

“It is now becoming clear through scanning technologies that the various senses

also share higher order cerebral networks, or perceptual supramodalities that

engage a crossover of sensory inputs from one sense to another”

- Harry Francis Mallgrave 2010

Over the past hundred years, increasingly refined theories of perception have

emerged. The top down philosophical theories are beginning to coalesce with the

bottom up scientific theories giving us greater insight into the study of experience1.

The relatively recent theory of Embodied Cognition has shed light on how the mind,

the body and the world form a conscious system of dynamic symbiosis. This recent

view of perception implies that experiences originally considered to reside solely

in the human mind such as music are in fact intrinsically connected to our bodies

and environment. Marc Leman describes this concept of embodied cognition:

“In contrast to dualism, the concept of mind is seen as an emerging effect of the

brain perceiving its own actions in relation to a physical environment.2 From that

perspective the subjective world of mental representations is not an autonomous

category but a result of an embodied interaction with the physical environment.”3 He

also suggests how our perception of music is tied to other embodied experiences:

“The multimodal aspect of musical interaction draws on the idea that the sensory

systems – auditory, visual, haptic as well as movement perception – form a fully

integrated part of the way the human subject is involved with music”. 3 This

dissertation will not focus on specific emotional processes involved in making and

listening to music. The primary aim of this discussion is to reveal and test how our

embodied interaction with the world develops perceptions of music and sound that

are fundamentally connected to our perceptions of space.

1 The history of phenomenology through to embodied cognition is described in further detail in Dourish (2004) p.102-1262 Ernelling and Johnson (2005)3 Leman (2008) p.13

Introduction.

5

To develop this hypothesis, I will examine the works of Iannis Xenakis, Oskar

Fischinger and Mark Applebaum. I will observe how these artists have been able to

transpose between sound, image and gesture by utilising and adapting the intrinsic

spatial inferences that music creates. These works will be examined based on the

principles of embodied cognition to find the theoretical basis for the vocabulary and

intersubjectivity of each medium. The discussion will also look briefly towards shared

neurological processes in music and space perception, particularly the narrative

and memory functions of the hippocampus necessary for both spatial navigation

and musical experience.

The research is consolidated with a series of spatio-musical interactions in the form

of volunteer tested experiments. The experiments use the theoretical framework set

out in the first chapters to actively engage the volunteer’s spatio-musical ability and

develop new kinds of intersensory interactions. I will then discuss the potential value

and opportunities given by this frame of mind towards architectural discourse.

For the purposes of this text I will use a broad definition of “music” as: any sequence

of sounds that are arranged within a larger temporal structure, and the term “sound”

to describe any audible form.

6

Listening to music is a vastly complex mental phenomenon that activates many

different parts of the brain. However, we can start to reveal the spatial aspects of

music perception by examining the ways in which we signify musical concepts.

The genealogy of music perception has consistently been related to the evolution

of language.4 Verbal communication, as well as refining the sophisticated pitch,

rhythm and pattern recognition necessary for music appreciation, uses higher level

representations to signify different experiences.

Linguistic descriptions of music (such as Hatten’s texts about Beethoven’s works

1994) rely heavily on metaphors to create intersubjective representations of melodies,

dynamics and timbres. These metaphors (such as Escalating, Swirling, Collapsing

and Sweeping) are not simply means to describe music but actively display how

we perceive music through the language of other experiences. George Lakoff and

Mark Johnson illustrate how our entire language is constructed from cross modal

inferences and that these associations form the essential rudiments in which we can

conceptualise and think about the world.5,6 They describe the metaphor of “affection

is warmth” as emergent from a child’s concurrent experiences of affection and warmth

in the embrace of a parent.6,7 This association between conceptual and sensory

inferences forms what Lakoff and Johnson call an “Embodied concept”.8 Our ability

to think is facilitated by both the brain and the nervous system7,8,9. The prevalence of

physical and spatial metaphors use to describe musical attributes and features, such

as dancing melodies, sweeping and swirling phrases, rising and falling scales etc.

allow us to see that our conceptions of music rely heavily on spatial understanding

and embodied concepts.

4 Brown (1999)5 Mallgrave (2010) p.175-180.6 Lakoff and Johnson (1980) p.255-2567 The neurological basis for metaphors in language is described by Lakoff and Johnson (1999) using Donald O. Hebb’s theory of concurrent sensory information and synaptic growth (Hebb 1949), This is explained fully in Mallgrave (2010) p.175-180.8 Lakoff and Johnson (1999) p.209 Tversky (2008)

Music Signification.

7

The words “high” and “low” used to denote audible pitch as well as vertical

position is a prime example of an audio-spatial metaphor that we can suggest a

possible etymology from an embodied perspective. Perhaps the most direct way

we interact with sound from the moment we are born is through the use of our

vocal chords. In producing or replicating different pitched sounds we must perform

certain biomechanical operations that are coordinated with our auditory sense.

Our vocal production of sound affects our spatial perception of pitch through the

perceived location of vocal noises relative to the ears. The position of the larynx

in the throat moves vertically to allow a higher or lower vocal range, this physical

change in position affects the perceived location of the produced sound. Singers

use the terms “Chest Voice”, “Middle Voice” and “Head Voice” to represent these

different positions as this is where the sound is felt to be coming from, this clearly

reinforces the perception that lower pitches are physically positioned below higher

pitches in space. This embodied explanation displays that the signification of

the words “high” and “low” to audible pitch is not an arbitrary selection of words

but is heavily reinforced by the spatial perception of our voices, catalysing this

association between pitch and height over time. This is one example of how an

embodied action produces cross modal inferences between seemingly disparate

sensory perceptions. This multimodal perception is consolidated and symbolised

as polysemous homonyms in language. This vertical mapping of pitch in space was

initially observed by C.C Pratt in 1930 after observing that the specific succession

of tones in a musical phrase can generate a sensation of vertical movement.10

These ideas have since been experimentally demonstrated by Lidji et al. 2007 and

Rusconi et al. 2006 and show the potential that audible properties can be mapped

successfully into spatial representations.11,12 I have highlighted just one example of

how spatial terms are used to describe audible characteristics in speech; however

the discipline of music has developed other means of signification to represent and

translate musical attributes in visual and spatial terms.

10 Pratt, C.C. (1930) 11 Lidji, P., Kolinsky, R., Lochy, A., & Morais, J. (2007)12 Rusconi E, Kwan B, Giordano B.L, Umilta C, Butterworth B (2006)

8

The musical score is a syntax and grammar used to translate music from a dynamic

temporal form to a static representation in 2 dimensions and vice versa. The 2

dimensions of space used in musical scores traditionally represent time read from

left to right in the x axis, and pitch arranged vertically on the y axis. The traditional

score clearly utilises the aforementioned spatial inference by signifying relative

pitches with vertical positions. The length of the note or rest is symbolised using

conventions of notation that must be learned in order to read the musical text.

Whilst the general form of instrumental music can be effectively transcribed with

these traditional graphical languages, the strict use of both spatial dimensions and

limited syntax of expression are abstracted from the experiential qualities of the

music13. This further asks the question whether there are alternatives to the standard

graphical musical signification that are able to capture the more experiential

qualities of music.

The composer Iannis Xenakis started to experiment with spatial representations of

sounds to reveal wider experiential truths within his music. His work Metastasies

originated with a graphical score that maintained the conventional use of

representational axis yet aimed to become more than just an instructional text for

performance (Fig. 1). Xenakis was particularly interested in the aspect of time in

music, the contemporary Einsteinian view had shown that time was relative to mass

and energy, in that any change in the contents of time would change time itself.

Xenakis aimed to emulate/explore this abstract idea in his music through the act of

capturing it in a static space.14 This negation of real time allowed the comprehension

of the whole in the instance of seeing the score. Xenakis used the example of

gunshots in a battlefield to describe the nonlinearity of these particular musical

13 Linguistic descriptions and conventional notations are critiqued by Bengtsson, I & Eggebrecht, H. (1973), and Leman, M. (2008)14 Hofmann, B. (2005)

The Musical Score.

9

Fig. 1 - Graphical glissandi from Xenakis’ Metastasis.

experiences; the exact order of each bullet fired is irrelevant as the resulting sound

as a whole will undoubtedly be that of “gunfire”. This concept of the musical form

being something other than a sum of its parts is particularly evident in Xenakis’

Mycenes Alpha (1978).The graphical score of Mycenes Alpha is comprised of many

(often straight) lines which collectively define larger, curved undulating masses (Fig.

2). The image that results is a collection of undulating conglomerate forms. The

graphical score was translated into audio by Xenakis’ own UPIC system15, whilst

the system translates each individual line into a note; the resulting cacophony of

all the combined sounds also generates the perception of an undulating dynamic

whole. Xenakis was fascinated by the form of the Hyperbolic Paraboloid (Fig. 3)

which too features this characteristic explored in his music: whilst every element

in a hyperbolic paraboloid is a straight line, the perceived whole is a twisting and

smoothly curved surface. These ideas were manifested spatially in the design of the

Philips Pavilion (Fig. 4). This is an example of how a perceived audial experience

has become effectively formalised to generate similar visual (and then spatial)

experiences. However, Xenakis was not the only proponent of the graphical score

and whilst his methods of representation reveal some ways in which the experience

of music can be perceived in static representations, others began to extend the

boundaries of how 2 dimensional spatial representations in fact become sonic

experiences in themselves.

Mark Applebaum’s Metaphysics of Notation (2010) rejects the standard

use of dimensional axis to represent specific attributes of sound and time.

His work instead relies solely on subjective associations and metaphorical

synaesthetic interpretations. This indeterminate notation does not specify any

instrumentation or temporal structure and thus, the individual performers of The

Metaphysics of Notation produce varying audial results. The musicians’ process

of transforming/analogizing the shapes and forms on the page into temporally

arranged soundscapes displays the brains ability to interchange sensory forms

to produce meaning. This ‘ability’ of multimodal chiasm is not simply a skill of the

conscious brain but is in fact a fundamental property of how the brain works.16

The scores themselves break so far from traditional scores that many of the

interpretations are completely unique, however some aspects of an intersubjective

15 The computerised UPIC system translated the vertical position of a mark into pitch and its horizontal length into duration.16 Mallgrave, H. F. (2010)17 Arnold, R. (2010)

10

Fig. 2Excerpt from Mycenes Alpha.

Fig. 3Example Hyperbolic Paraboloid.

Fig. 4Philips PavilionXenakis & Le Corbusier.

associative language are revealed by intrinsic similarities between the performers’

interpretations of certain passages of shapes.17 Some of these are simply carried

across from the performers’ traditional training of scores; sweeping and curving

lines become long notes changing in pitch respectively, and sections of repetitive

spots seem to unanimously represent individual notes or rhythmic musical events

in time (Fig. 5). Some similar responses that aren’t traditional of standard notation

are that the size of the elements are commonly attributed to volume, this is used in

the standard crescendo symbol but is also a basic perception that larger objects

generally make louder sounds when interacted with. Associations derived from

common perceptual experiences of actions and sounds in the world form a key

aspect of the sensory chiasm that allows the translation of these images into sounds.

Shape, texture and formal characteristics are also passed between our experiences

of sound and sight; they are intrinsic to the legibility of the graphical scores and

have been observed experimentally.

The Kiki/Bouba effect first observed by Wolfgang Kohler 1929 gives great insight

into how shape and formal characteristics of objects produce strong inferences to

types of sound.18 These synaesthetic connections are essential subconscious tools

that the musicians employ when translating the many shapes of “The Metaphysics

of Notation” into music. Kohler tested this idea by presenting subjects with two

shapes, a smoothly curved anamorphous blob and a jagged, angular star-like shape

(Fig. 6). When asked which shape was called “Kiki” and which was called “Bouba”,

95% of subjects assigned “Kiki” to the spiky form and “Bouba” to the rounded blob.

Speculatively this could be connected to the visual graphemes used to represent

the verbal sounds. For example the letters of the word “Kiki” feature sharper, more

angular lines than those in the word “Bouba”, which ubiquitously features rounded

letters. This implies that there may be some aspect of reading the shapes in the

graphical score using the musician’s functions of verbal language, developed in

parallel to their musical training. However, further work on sound-shape correlations

by Daphne Maurer in 2006 observed that the Kiki/Bouba effect was also prevalent

in toddlers as young as 2.5 years old.19 Whilst the effect was slightly less prominent,

the fact that the toddlers had yet to develop reading skills shows that the shape-

sound inference is primarily based on the audible sound rather than inferences back

to a visual representation of the words. Ramachandran and Hubbard hypothesised

18 Köhler, W. (1929)19 Maurer D, Pathman T & Mondloch C.J. (2006)20 Ramachandran, V. S., & Hubbard, E. M. (2001)

11

Fig. 5

Fig. 5 - Excerpt from The Metaphysics of Notation

Fig. 6Kiki. Bouba.

that this association is built through the use and shape of the mouth in creating

the sounds of the nonsense words: the angular figure “mimics the sharp phonemic

inflections of the sound kiki, as well as the sharp inflection of the tongue on the

palate.”20 From this we can see how our ability to translate shapes into sounds

is also derived from an embodied application of using our mouths and lips to

produce different sounds. However, many musicians have refined interactions with

sound (other than using their vocal chords) that they use to interpret Applebaum’s

scores. For these examples we must look at the coupling of body to instrument,

and then to the affordances of the instrument in terms of the corporeal motion used

to produce sounds. The limitations and facilitations of the instrument will invariably

affect the interpretation of the visual form. The bodily motion that changes the

pitch of a trombone for example is very different to that of playing a cello or flute;

as such, associations of movement across the score could yield different audible

results. The empathic association that allows the symbols to be read as sound

generating movements is another factor in interpreting these scores, and can be

generated through the very act of drawing the score. The signification of the musical

expression has emerged from a motion, such as a sweeping brush stroke or flick

that has imprinted its temporal history into a static symbol. The intentionality of the

artist in interpreting the symbol can be read through their empathic comprehension

of how the mark was made, this can then be manifested as corporeal musical

expression.21 We have seen how forms on a 2 dimensional plane can infer audible

responses, through convention, “embodied concepts” and even inferences to

motion and gesture that inform the instrumental output. This inferred motion and

mental animation of the symbols on the score as they are interpreted displays the

limitations of 2D graphical representations. Music is dynamic and it appears that our

spatial representation and cognition of music can be further augmented through a

dynamic spatial medium.

21 Corporeal imitation, articulation and expression are further investigated in Leman, M. (2008) p.103,123,141

12

The medium of film and in particular the animations of Oskar Fischinger address

some of the dimensional abstractions of graphical scores as the visual space and

animation occupy the same temporal dimension as the music itself. Whilst the

visual metaphors between shape and sound such as size to volume and shape

to timbre etc. are still used to great effect in Fischinger’s films, the possibilities of

movement are released by the animated medium. Musical objects now exhibit

speed and characteristics of movement that bring with them their own physical

and aural associations. Objects that exist within the dimension of time (unlike the

images of Applebaum and Xenakis) can swoop, shake, dart, fold in on themselves

and perform a wide variety of dynamic behaviours; as a result the animations get

closer to synthesising these ‘intuitive’ connections between sound and visual form.

Susannah Knights writes of Oskar’s animations: “Both media unfold and engage the

audio and visual senses over time, similarly capable of evoking an automatic sense

of narrative expectation. Secondly, both create an illusion of a gestalt, through the

audience psychologically connecting component parts which run in a temporal

sequence.”22 The dynamic properties of Fischinger’s films reveal how the viewer’s

expectations of imminent musical and visual activity can become synchronised to

further enhance the perception of an audio-visual gestalt. Our ability to expect future

events based on current conditions is based on observing recurring phenomena

to build an understanding of how objects behave. If we continue to use embodied

experiences and interactions as a basis for generating audio-visual associations

Animated Music.

22 Knights, S. (2012) p.13

13

we can start looking at our spatial expectations and see how these interact with our

musical expectations. For example, the resolution of objects in motion is for them to

fall downwards under gravity and come vertically to rest. We use this expectation

and understanding of the mechanics of our environment to be able to perform

actions, such as catching a falling ball. The same kind of resolution also appears

in chord sequences. Sustained chords and perfect cadences also create a tension

that is traditionally expected to be resolved. This effect of expected resolution is

evident in our melodies of speech, for example an upwards inflection at the end of a

sentence or melody creates a musical question that expects an answer. Susannah

Knights observes how Oskar exploits this sense of melodic narrative and physical

understanding: “although the first two strokes arch downwards on the audible

downbeats, the third, marking a quaver leading-note which sparks a short passage

of syncopation, arcs upwards, briefly creating a similar expectation of a resolution.”23

(Fig. 7) This expectation is not only built with the motions that precede it, the visual

and audible upward inflection evokes the same expectation of resolution as a verbal

question or object being flung into the air. We can see that, as static shapes can

elicit audible timbres, we also perceive motions and expected trajectories of both

dynamic physical objects and musical phrases.

23 Knights, S. (2012) p.17 describing Fischinger’s Studie No. 7 00:37 † Images are 20 frames of animation superimposed to show motion paths.

14

Downbeat 00:37

Fig. 7†

Downbeat 00:38

Expectation 00:39

Resolution 00:40

Fischinger’s films visibly correlate to our embodied perception of music’s motions

and dynamic patterns far more effectively than static graphical symbols. However,

both mediums achieve their audio-spatial inferences through embodied concepts

and common sensory experiences. Mark Applebaum’s piece Aphasia (2010)

combines hand gestures with music to create an audiospatial gestalt that is created

around the body itself. The medium of hand gestures not only moves the experience

of the music into a dynamic 3 dimensional realm but immediately generates an

implication of action to the audience. We use our hands as a primary means to

interact with the world, therefore a movement of a hand implies an action is taking

place in which there will be an effect. Unlike the dancing shapes and sprites

of Fischinger’s films, that are simply characterised by motional behaviours that

complement the music, the movements of the hands are perceived to have a direct

causal relationship to the music that they accompany. The electronically altered

vocal samples and abstract audio snippets used to create the audio of the piece

do not remind us of any traditional instrumentation, however the hand gestures

appear so perfectly matched to the audio that it is difficult to imagine which is

derived from the other, or if they were composed in parallel. Applebaum’s advanced

and multidisciplinary instrumental experience has given him a sophisticated

understanding of how motions and hand actions produce sounds with objects. His

search for new musical interactions has led to the invention of bespoke instruments

such as the “Mousketeer” (Fig. 8). Applebaum avoids the motional associations that

accompany traditional instrument sounds (such as the smooth sweeping motions of

the string instruments visualised to great effect in Fischinger’s Studie Nr. 7 01:22).

Instead he creates an original vocabulary of sounds and motions that is strictly

adhered to in the performance of Aphasia. The lack of a perceptible time signature

augments the audience’s perception that the sounds are causally linked to the

actions as they appear spontaneous yet temporally synchronised. By repeating the

sound gestures, the audience signifies the visible motion to the audible response

Sound Gestures.

15

Fig. 8 - Mark Applebaum playing his “Mouseketeer”.

and gradually learns the vocabulary of the piece as it progresses and new sounds

and motions are revealed. By producing the illusion that the motions actively create

the sounds, the actions that Applebaum uses appear to imbue the sounds with a

physical presence and spatial attributes. The piece opens with several disparate

beats to the chest, creating a short percussive knocking sound. After the sixth

instance of this action, instead of returning his hand to his lap his arm opens out

away from his body (Fig. 9 - Aphasia.mp4 - 00:34), the knock appears to echo

around this new space that the gesture has created. This perception localises the

sound of the knock to the space around the body that is perceived to be its source.

Similarly at (Fig. 10 - Aphasia.mp4 - 02:12) Applebaum strenuously pulls his hands

apart emulating a tension that is audibly reflected with a series of strained rubbery

squeaks. The perception that the sound is being tangibly stretched apart is a very

successful example of the sounds assuming a spatial and tangible form in the mind.

It must be noted at this point that a core aspect of embodied cognition is that for

any intentionality or experience to be embodied it must also be situated. “People’s

conceptions of space differ for different spaces and are a joint product of perception

and action appropriate for those spaces”24 This is the idea that behaviour is situated

in it’s architectural environment.25 A person might be more likely to dance if music is

played in a dance studio however would perhaps act differently in a church or office.

The gestures of Applebaum’s Aphasia are not only afforded by the biomechanical

trajectories of the human body but also influenced (or indeed generated) by the

wider spatial context and the perception and action appropriate for the environment

itself. The piece is primarily viewed on a blank stage or in front of a black

background as shown in the filmed version. There are no visual clues to the spatial

context in terms of architecture, however the piece is by nature a performance so

there is an intrinsic factor of ‘Audience’ that informs the spatial representations of the

music that Applebaum creates. This audience is either a real audience in the stage

context or the camera in the filmed context. This directional connection between

audience and performer changes how the gestures are interpreted, it would

certainly not have the same effect if viewed from behind. We can see how the study

of spatio-musical chiasm in a dynamic 3 dimensional space demands embodied

and situated means of analysis; the spatial context and relative perspectives of

people within the system deeply affect the resulting musical interpretation.

24 Tversky, B. (2008) p.20225 Behaviour and language are spatially situated but not environmentally determined Dourish, P. (2004) p89

16

Fig. 9

Echo Gesture.

Fig. 10Stretch Gesture.

By initially looking at examples of prior art we have seen how static and dynamic 2

and 3 dimensional spaces have been used to naturally mediate our perceptions of

sound, music and space. I have also described how these gestalts are formed in

the mind by utilising the approach of embodied cognition26,27 and the action oriented

ontology of music described by Leman. The precedents have not only examined the

spatial inferences of individual sounds, but also touched on how the sounds (and

indeed shapes and gestures) in the music relate to each other in time and generate

expectations, moments and contrasts that are attuned to our embodied experience

of the physical world. This aspect of music distinguishes it from being simply

sequences of sound or noise and deserves greater examination as it displays more

fundamental ways in which space and music are shared in the brain.

26 Tversky, B. (2008) 27 Shapiro, L. (2010)

17

Marvin Minsky’s description of music as an experience of learning begins to illustrate

how the temporal structures of music are experienced and formed in the mind. He

uses the words “sonata as teaching machine”28, describing the traditional musical

sections of ‘exposition’, ‘development’ and ‘recapitulation’ as phases of observation,

assimilation and accommodation in learning how to listen to the rest of the music29.

In a simple sense the gradual exposition of initial rhythms, harmonies and scales

start to build a psychological temporal framework of time signatures and key

signatures to which the rest of the music can develop a relationship. Minsky makes

the distinction that “Learning to recognise is not the same as memorizing”30 and that

whilst we rarely memorise entire pieces of music, something remains in the mind

that allows phrases to trigger memories of the subsequent bars. He illustrates the

similarities between our use of sight and of music in building mental maps: “How do

both music and vision build things in our minds? Eye motions show us real objects;

phrases show us musical objects. We “learn” a room with bodily motions; large

musical sections show us musical “places.” Walks and climbs move us from room

to room; so do transitions between musical sections. Looking back in vision is like

recapitulation in music; both give us time, at certain points, to reconfirm or change

our conceptions of the whole.”31 This musical view of space not only applies to

“rooms and objects” but also has resonances with the views of navigation in larger

scale psychogeography. In navigating a city or country route, we do not memorise

the exact spatial arrangement of our environment, however moments, contrasts

and landmarks along familiar paths cue us in understanding how our current space

relates to other nearby spaces (Fig. 11). Minsky suggests that the parts of the brain

that allow us to map spaces in this way are the same ones used to appreciate

musical form. This leads to the hypothesis that our experience of music uses similar

mental processes of learning and cognition to those of spatial navigation.

28 Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.2929 Stages of learning outlined in Piaget. (1970)30 Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.3031 Minsky M. (1981) In Clynes M, ed. Music, Mind and the Brain: The Neuropsychology of Music p.37-38

The Psychogeography of Music.

18

Fig. 11 - Front cover of Debord’s “Guide Psychogeographique de Paris” illustrates mapping of ‘the city’ based on the relationships between spaces, not actual positions.

Cupchick et al tested the neurological aspects of this hypothesis in their paper

“Shared processes in spatial rotation and musical permutation”.32 The paper found

a direct correlation between a subject’s performance at a spatial rotation task and

their ability to detect whether a tune had been played backwards or inverted etc.

Cupchick’s paper illustrates the mental chiasm used to manipulate physical and

musical objects; this is neurological evidence of a mediating process that facilitates

our experiences of the precedents in the first chapter. However, these tests are

primarily focussed on short length musical phrases and technical cognition such as

pitch processing. They do not illustrate the wider temporal experiences of narrative,

expectation and contrast that are intrinsic to musical experience. The hippocampus

has been described as the essential area of the brain that allows the perception

of temporal and spatial narratives. “In addition to its outstanding role for memory

and spatial navigation,33,34 the hippocampus has been suggested to be involved in

novelty detection 35,36. Hippocampal novelty detection is based on a comparison of

actual sensory inputs with stored stimulus patterns.37,38,40”. 41This is clear evidence

for Minsky’s learning analogy of music. For example listening to a preliminary bar of

music is a “sensory input” that becomes a “stored stimulus pattern” in memory, as

the next bar is heard this new “sensory input” is compared to the growing structure

of the previously heard bars by the hippocampus. This is also a neurological basis

for the aforementioned expectation generated in Oskar Fischinger’s Studie No.

7. This emerging structure grows as we are exposed to more of the music, and

the subtle or substantial harmonic shifts and changes push and pull us in varying

degrees of expectation42. The hippocampus in this sense facilitates our perception

of rhythm, flow, continuity and surprise. This narrative experience separates what I

have called “sound” from “music” in the same ways that a single frame of a film is

different to the film itself, or a photograph of a street is different to walking through it.

It is now known that taxi drivers and musicians both develop larger hippocampi due

32 Cupchik, G. C., Phillips, K., & Hill, D. S. (2001)33 Maguire, E.A. (2001)34 Ekstrom A. D, Kahana M. J, Caplan J. B, Fields T. A, Isham E. A, Newman E. L, Fried I. (2003)35 Knight, R. (1996)36 Strange B.A, Fletcher P.C, Henson R.N, Friston K.J, Dolan R.J. (1999)37 Gray J.A, Rawlins J.N.P. (1986)38 Strange B.A, Dolan R.J. (2001)39 Vinogradova, O.S. (2001)40 Kumaran D, Maguire E.A. (2007)41 Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B., & Cattapan-Ludewig, K. (2010) p.142 Huron, D. (2006)

19

to their practiced experience and reliance on both spatial and temporal narratives

and memory. 43,44,45,46 The increased hippocampal neuroplasticity clearly illustrates

the essentiality of the hippocampus in perceiving narrative and temporal contrast,

and thus in facilitating our higher level emotional responses to sensitively curated

temporal experiences.

This associative understanding of human navigation through space is currently in

common use by architects and urban planners to generate consistent elements

and extensions of urban form. The reason for this is not neurologically based

but has evolved as a refined human sensitivity to city making. Our experience of

the city is built on a framework of how spaces relate to each other. We learn the

relationships between roads, streets, alleys and open spaces as we move through

the city, these relationships are used as a subconscious mental framework that

we use to comprehend and operate within other cities and spaces. By examining

the processes and functions of the hippocampus, we have seen that music and

dynamic navigation of space share this cognitive process, yet the parallels between

sensitive urban design and musical compositions are rarely consciously used.

Many metaphors such as “harmony” and “rhythm” are used to discuss and design

architectural proposals and their relationships to their context, yet few actively

engage the processes of musical composition to augment the embodied experience

of the city that they are developing. Perhaps by further understanding the deeper

complexities of music and its interaction with mind, body and place we could use

Music Aided Design (MAD) to develop deeper and more emotionally engaging

experiences of architecture and the built environment. This specific area of spatial

musicality deserves a lot more research but we are beginning to see how musical

experiences and spatial experiences are not as distinct as previously thought.

43 Maguire, E.A. Gadian D.G, Johnsrude I.S, Good C.D, Ashburner J, Frackowiak R.S, Frith C.D. (2000)44 Gaser, C., & Schlaug, G. (2003)45 Rodrigues, A. C., Loureiro, M. A., & Caramelli, P. (2010)46 Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B. & Cattapan-Ludewig, K. (2010)

20

We have examined how recent research and development in neuroscience, music

cognition and embodied cognition provide a framework for a spatial interaction

based ontology of music, this has been illustrated through basic observations

about how our human physiology affords particular experiences of the world and

also by examining similarities in neurological processes. A series of 6 interactive

experiments were developed in order to directly expand upon the multimodal

experiences described in the first chapters. The information and precedents

discussed suggest that we should also be able to form a symbiotic perception of

space and music that uses our knowledge and experience of both to produce an

intuitively interactive gestalt.

The 6 experiments featured 3 focussed on “sound” and 3 focussed on “music”

and were followed by a short questionnaire about the different interactions and the

user’s experiences (Fig. 12-13). The experiments were tested on students ranging

in musical ability and spatial awareness and were placed in a category of either

“Architect” (architecture students with less than 5 years musical experience),

“Musician” (non architecture student with over 5 years of musical experience),

“Musician Architect” (Architecture student with over 5 years musical experience)

or “Non Musician” (does not study architecture and has less than 5 years musical

experience). The main body of the questions required an X on a spectrum between

Agree and Disagree; the data of each questionnaire was calculated by measuring

the distance on the spectrum and dividing it by the length to create a coefficient of

agreement (0 represents strong disagreement and 1 represents strong agreement).

(Film clips of the experiments are on the Data DVD at the back)

Experimental Research.

21

Fig. 12 - Experiment Setup.

Fig. 13 - Speaker placement and Camera field of view.

Body Tracking CameraParticipant

Custom Software

Surround Audio

The first experiment the user was introduced to aimed to test the prevalence of the

embodied ‘Pitch is Height’ metaphor previously examined by C.C Pratt et al.10 The

user was effectively presented with an invisible plane of sound that was consistently

at arm’s length in front of them (Fig. 14) (PitchAndHeight.mp4); as the user extended

their arm out into this field an audible sine wave was produced. The frequency of

the sine wave was directly related to the user’s vertical hand position such that when

the user moved their hand up, a higher pitch was produced and vice versa a lower

position produced a deeper tone. The distance the user reached out correlated to

the sound’s volume, emulating the effect of force or pressure in creating the sound.

The pitch and volume would correlate to their hand’s movement and position in

real time. After exploring the relationships between sound and space for 1 minute

the relationship was reversed such that moving the hand towards the ground now

produced a higher pitched tone. The users were given another minute to play with

this interaction and were asked which interaction they found more intuitive and

natural. 92% of the subjects agreed that the deeper tones belonged below the

higher pitches in space and the average coefficient of agreement was 0.84. One

“Architect musician” subject felt very strongly that the higher pitches intuitively

belonged below the deeper pitches in space, it was soon made evident that this

particular user had played the cello for 12 years from the age of 6, the embodied

action involved in raising the pitch of a cello indeed requires a lower hand position

in space. This demonstrates the hypothesis that relationships between sound and

space are forged through corporeal interaction and experience.

Pitch and Height.Experiment 1

22

Fig. 14 - “High” and “Low” Frequency Sound Plane Visualised.

The second experiment used a similar mechanic of reaching away from the body to

produce a sine wave pitch yet the responsive area was arranged radially around the

center of the body as a ring at arm’s reach. The Highest pitches pointed towards the

front of the lecture room and the lowest pointed towards the rear of the lecture room.

The users were able to tap into this audial sound compass by extending their hands

out and feeling for direction with the sound (Fig. 15). The users were not told how

the sounds were oriented but were given 2 minutes to explore their affective audible

environment. The users were then blindfolded and disorientated by revolving on

the spot a number of times. The blindfolded, disoriented users were asked to

simply point towards the lectern at the front of the room. Despite only experiencing

this interaction for 2 minutes, 88% of the subjects were able to successfully and

instinctively use the sound as an audible sense of orientation and point in the correct

direction. This effectively shows how sound information can be imbued with spatial

orientation information and that even with only 2 minutes of exposure to the system

the users were able to use their augmented senses of proprioception and hearing to

make confident spatial decisions about their orientation in the space.

Sonic Compass.Experiment 2

47 The speakers used for this experiment completely surrounded the user such that no sense of stereo-graphic orientation could be used

23

Fig. 15 - Reaching into Sound Compass.

The third experiment featured three

differenty pitched string instrument sounds

that were hidden within the room. The

sounds were modelled as spheres that

emitted a continuous note when occupied

by a hand. The volume of the sound

increased as the hand moved towards

the center of the sphere. The users had

to explore the space at different heights

until they made contact with one of the

sound volumes (Fig. 16). This interaction

relies solely on the connection between the

aural sense and the sense of the body’s

position through proprioception and sight.

The users were surprised to discover the

sounds at first but went on to discover the

others using various methods of scanning

the space (Fig. 17). 63% of the subjects

experienced the sounds they interacted with

as 3 dimensional forms in space, however

the perception of the actual shape of the

sounds in the air ranged from the accurate

description of spheres through to columns,

discs, walls and cubes. The experiment

was modelled such that the sounds were

not only different heights but varied in

radius between 30cm and 50cm, 67% of the

subjects perceived the change in physical

size of the sounds in the space. 83% of the

subjects were able to correctly relocate the

position of the invisible sounds in the room

and could even play them as chords using

both hands.

Sounds in Space.Experiment 3

Fig. 16 - Searching and Finding Sounds (Film Stills)

This ability to map the experience of a sound into a 3 dimensional position through

bodily interaction illustrates the potential of sensory chiasm when perceiving spatial

objects and sounds. The aural connections between sound and shape discussed in

the first chapter imply that the perceived shape of the sound would be affected by

the timbre and texture as well as just the interactive spatial experience. In the future,

this experiment could be expanded by using different types of sound as well as just

different positions, pitches and sizes to test if perceptions of shape and form were

influenced by audible timbre.

Fig. 17 - Searching and Finding Sounds (Sounds visualised)

24

24

Fig. 19 - Correct Zone Diagram

Fig. 18 - Visualisation of 3 Continuous Sound Zones

Fig. 20 - Incorrect Zone Diagram

Photoshop image here!

Experiment 4 tests how the presence of a modulating sound field can change a

user’s perception of a room as they move through it. The room was divided into 3

overlapping zones of music, such that as the inhabitants moved around the room,

the audible characteristics of the space changed with their position (Fig. 18). The

3 pieces of music had their tempo synced with each other such that there was a

smooth transition between each piece when the users moved between zones. The

users explored the room to discover the new audible significance of the different

spaces, most of them developed new preferences for particular positions in the

room. 96% of the users found that the music significantly changed the character of

the spaces, and 96% were able to perceive the spatial presence of a modulating

sound field. The experiment also caused some users to jump from one zone to

another at the end of a bar to further control the music as well as passively observe

the changes. The questionnaire required the users to draw a map of the sound

spaces on a floor plan of the room. 67% of the users accurately illustrated the

spatial layout of the music zones (Fig. 19). The remaining participants produced

less accurate diagrams of how the room was divided, yet still picked up some

attributes of the zones such as the fact that the zones overlapped (Fig. 20). Only

54% of all architecture students we able to successfully illustrate the zone divisions,

whereas the other 46% had close representations but generally overcomplicated

the spatial arrangement; this suggests that the architect’s refined skill for thinking in

plan had not necessarily expanded their ability to map the sounds into space but

perhaps added extra complexity from other factors in the music. However, 70% of

the musicians were able to draw the zones in plan with near perfect accuracy. This

is evidence that instrumental experience builds and reinforces strong associations

between corporeal action in space and audible reaction.

Spaces of Sound.Experiment 4

25

Fig. 21 - Walking through music, forwards and backwards.

The 5th experiment aimed to bring the temporal dimension of music into a physical 3

dimensional representation that the participants could interact with. The experiment

was developed by again looking to metaphor as a source of crossmodal chiasm,

particularly the words we use to talk about direction and time. We use the terms

such as “looking forward to something”, “backwards”, “behind” and “thinking

ahead” to talk about linear directions in time. Tversky describes the embodied

meaning of the word “forward” as the space in front of our bodies, or our primary

direction of movement.48 Using the principle that this linguistic metaphor has both

temporal and spatial associations, the experiment was developed such that any

forward motion of the user (such as walking) would cause the music to play; if

they stopped, the music stopped, and if the user walked backwards, the music

would also play backwards (Fig. 21). Through interacting with the space the users

were allowed to feel the alignment of their motion with the dimension of time and

therefore have the ability to move through or control it at will. 96% of the participants

agreed that the music moved in the same direction as their body with the strong

average agreement of 0.88 (MusicAndMotion.mp4). A common response to this

experiment was that the subjects tried to control the tempo of the music with how

fast they moved, this feature was omitted due to the technical challenges and time

constraints, however future versions of this experiment could be greatly enhanced

by allowing the subject to travel along the temporal dimension of music at their own

speed within the space as well as simply direction. These comments and statistics

of engagement clearly illustrate that we perceive both music and spatial motion as

dynamic temporal structures.

Music and Motion.Experiment 5

26

48 Tversky, B. (2008) p.201

Music, sound and noise are commonly experienced

as ubiquitous qualities of space. The sixth experiment

aimed to generate the perception that the music

playing in the space could be physically held and

manipulated with the hands and body. To generate

this perception of a physical and interactive sound

field, the actions that altered the sound were

designed to harness common sensory experiences

and associations to create an immediately intuitive

interface. The experiment focussed on the character

of muffling and dampening sounds and the spatial

factors that create this audible change. The sensory

ubiquity of sound and noise in the everyday

spaces we inhabit has built a comprehensive

perceptual understanding of how sounds act

within and permeate different environments. For

example the acoustic responses of sound in large

and small spaces have immediately recognisable

characteristics of reverb. This relationship also

allows artificial or reproduced sounds with greater

reverberation times to strongly elicit senses of

spatial scale without visual cues (such as the

aforementioned echo gesture in Aphasia). A similar

common experience of sound behaviour is the

natural muffling of sounds in enclosed spaces.

Obstacles between a listener and a sound source

will absorb and reflect different frequencies of the

audio depending on the form, depth and materiality

of the obstacle. For example, building walls tend to

transmit lower audio frequencies as their wavelength

exceeds the depth of the obstacle; as such, music or

noise from a neighbouring room with poor acoustic

Haptic Music.Experiment 6

Fig. 22 - Screenshots of “Haptic Music” program.

insulation will appear muffled and

will lack the definition of the higher

frequencies. From this common

experience, muffled sounds carry

with them associations of spatial

conditions such as being enclosed

in a box, underwater or hidden

behind a wall. The experiment

aimed to harness this spatial

association of muffled sounds

to generate a perception of a 3

dimensional source of pure music

that could be held in the hands,

gathered, enclosed, squeezed,

compressed and even released

back into the room. The interaction

was devised such that as the hands

of the user pressed together a low

pass frequency filter would act on

the music playing in the room, the

cutoff frequency of the filter was

directly proportional to the proximity

of the hands providing the illusion

that the hands were enclosing the

source of the sound. At the point at

which the hands came completely

together, the music had been

reduced to a deep, compressed

throb. If the hands were unclasped

rapidly the music appeared to have

“escaped” and all frequencies were

returned to normal. To capture the

music again the subject had to

extend their arms out to “gather”

the audio and then the music

was able to be compressed and

released at will.

Fig. 23 - Collecting and Releasing Music (augmented video stills - HapticMusic1.mp4).

27

In this experiment the subject was asked to stand in the middle of the space whilst

the music began playing through the speakers. The subject was then simply asked

to “compress the music” but was given no visual or gestural cues of how this

might be done. The verbal cue to “compress” the sound elicited various different

responses from the subjects, some extended all their limbs before crouching

and curling into a ball (Fig. 24), others raised their hands up then pressed them

together towards the floor and most of the subjects brought their hands together as

if squeezing a large ball (Fig. 25-26). These initial responses indicate the success

of the application of a spatial metaphor to the manipulation of the music. As the

subjects realised that their initial motions affected the music in an expected manner

they almost immediately began to interact with the music as an object or field in

space. 96% of the subjects agreed that the interaction between body and music

felt natural with a strong average agreement of 0.89. The experiment was not only

devised to test whether a spatial sense of music could be achieved through intuitive

interaction but also to test how the temporal structure of the music would influence

the subject’s choice of movement once the perceptual coupling between action and

effect had been made.

Haptic Music (Continued).

Fig. 24 - Crouching to hold the music (Video Stills).

Fig. 25 - Compressing to hold the music (Video Stills).

The music the subjects manipulated was chosen specifically to encourage

participation; the music featured a strong underlying rhythm with a catchy repetitive

chord progression and melody.49 The contemporary music used had very clear

dynamics, build ups and releases at the end of regular 16 bar phrases. The

repetitive nature of the beat and chord progression was chosen in order to introduce

the subjects to the temporal framework of the music quickly such that they were able

to comprehend and act on their expectations of change by reacting with appropriate

body movements. By compressing the sound and reducing the music to the

bassline and underlying beat, the subjects were able to contribute to the dynamic

changes of the music, generating a literal tension and anticipation of resolution that

is felt both musically and physically. This is a similar effect to the stretching action

and audio used in Applebaum’s Aphasia. 83% of the musician subjects agreed that

the structure of the music influenced their choice of motion, yet only 40% of the non-

musician non-architect group found this to be the case. The musicians had a clearly

refined awareness of corporeal action with regards to musical structures. Whereas

the primary aim of the non-musicians’ actions was simply to examine the effect of

the resulting tone of the sound. 100% of the Architect musician group agreed that

the structure of the music influenced their choice of movement with an average

agreement of 0.93. This clearly illustrates a relationship between the subject’s

musical and spatial experience and their ability to think in 4 dimensions.

49 The music was of a popular genre to the subject demographic yet not well known enough to distract the subject from the spatial interaction

Fig. 26 - Experiment photograph with overlaid illustration of sound.

28

28

Table 1. - Experiments Overview.

• 92% of participants placed higher pitches above lower pitches.

• Some musician’s instrumental experience had reversed this placement.

• Spatial associations to pitch are built through embodied actions.

Experiment 1:

Pitch and Height

• 88% of participants orientated themselves using proprioception and hearing.

• Participants could functionally use spatial arrangement of sounds to make decisions.

• Sounds are be imbued with spatial meaning over time and experience.

Experiment 2:

Sonic Compass

• Spatially interacting with sounds can perceptually give them a 3 dimensional form.

Experiment 3:

Sounds in Space

• Audible attributes of a space significantly change our experience.

• Sound fields can be perceived spatially (and accurately described).

Experiment 4:

Spaces of Sound

• We feel time as moving forwards.• Music can become a navigable string of

movement in space.

Experiment 5:

Music and Motion

• We can feel the spatial qualities of music and sound with our ears.

• We can use previous understanding of how sound works in space to manipulate sound and music as physical objects.

Experiment 6:

Haptic Music

29

Expe

rimen

t >1

25

Que

stio

n N

umbe

r >1

23

45

67

89

1011

1213

M/F

Mus

icia

nSu

bjec

t1

23

45

6Su

bjec

t Num

ber v

160

5960

4560

6060

585

5958

5760

F1

Arch

itect

ure

43

26

51

253

5430

1126

4657

475

4456

5455

M1

Phys

ics

45

62

31

360

6030

1545

1560

4317

5815

3050

M1

Engi

neer

ing

61

32

54

450

3760

6032

5757

600

5160

5951

M0

Arch

itect

ure

56

32

41

546

030

3047

4346

460

4430

6042

M1

Hist

ory

65

12

43

657

5730

5240

4030

5042

1554

5052

F1

Chem

istry

53

24

61

759

5959

5959

4242

592

3347

4546

M0

Arch

itect

ure

56

42

31

860

510

4660

5960

600

6030

5916

F0

Arch

itect

ure

62

34

51

934

1050

6058

6060

600

6011

6060

F0

Hist

ory

of A

rt4

53

26

110

059

3044

6060

6060

060

6060

60M

1Ar

chite

ctur

e5

64

23

111

5047

3426

3456

3347

257

5757

58M

0Ar

chite

ctur

e4

36

52

112

6060

6060

4360

6060

060

6060

60F

1Ar

chite

ctur

e2

56

34

113

1537

4517

3747

3640

1545

4050

54F

1Ar

chite

ctur

e2

51

46

314

5858

5858

2155

5530

056

5656

56M

0Ar

chite

ctur

e2

45

36

115

608

5052

3050

5050

650

6060

60F

0Ar

chite

ctur

e5

64

32

116

5460

6024

4060

6060

060

3060

60F

0Ar

chite

ctur

e5

64

13

217

6045

6060

6060

4560

752

6060

60M

1Ar

chite

ctur

e2

64

53

118

3760

5459

5959

5959

1242

5960

60F

1Th

eolo

gy5

61

23

419

6060

6060

4057

5960

060

4660

59F

0Ar

chite

ctur

e5

36

24

120

5745

442

4760

5755

457

1154

50F

0PP

S5

46

13

221

5545

4712

4060

6058

555

1352

30F

0Hi

stor

y5

64

13

222

5545

5554

5444

4440

1245

5356

45M

1En

gine

erin

g5

63

24

123

5052

1330

5052

5360

560

5853

53F

0PP

S3

46

52

124

5540

340

3060

4949

1048

5253

53F

0Bi

olog

ical

Sci

ence

s5

46

32

1

Aver

age

rank

of p

refe

renc

eAg

reem

ent C

oeffi

cien

t0.

840.

770.

680.

710.

740.

880.

870.

880.

100.

850.

750.

920.

876

45

31

2Sa

mpl

e siz

eAr

chite

cts

0.94

0.79

0.79

0.80

0.66

0.91

0.87

0.89

0.02

0.89

0.80

0.95

0.85

86

45

32

1M

usic

ians

0.86

0.77

0.64

0.61

0.75

0.69

0.82

0.79

0.24

0.69

0.74

0.86

0.84

64

63

25

1Ar

chite

ct M

usic

ians

0.65

0.87

0.85

0.75

0.87

0.96

0.87

0.93

0.09

0.92

0.93

0.96

0.98

56

31

45

2N

on E

ither

0.84

0.64

0.39

0.61

0.75

0.97

0.93

0.94

0.08

0.93

0.48

0.91

0.82

56

45

12

5

% A

gree

men

t92

8863

6783

9696

964

9675

9692

Succ

essf

ully

orie

ntat

ed

Arch

itect

s10

088

8875

7510

010

088

010

075

100

8863

Mus

icia

ns10

083

3350

8383

8310

017

8383

8310

083

Arch

itect

Mus

icia

ns60

100

8080

100

100

100

100

010

010

010

010

040

Non

Eith

er10

080

4060

8010

010

010

00

100

4010

080

60

34

6Ge

nera

l

Table 2. - Experiment Data.

The results of the experiments clearly show that we have the ability to perceive

both music and sound as spatial forms. We naturally use metaphors to mediate

meaning between musical and spatial experiences yet rarely actively engage our

experiential knowledge of one with the other. This ability is not reserved to those with

neurological synaesthesia, it is built through our everyday experience. It is clear that

we have moved beyond the question “Are music and space cognitively connected?”

to “How can we apply this unified understanding of embodied perception to

design?” and particularly “How can the diciplines of music and architecture

augment and inform each other?”

I have demonstrated that by using emerging technologies and systems, both

spatial and audible presence can be combined to augment a user’s navigation and

experience of architecture. The results of the experiments speculate that these forms

of navigable music could become an architecture unto themselves. In experiments 3

and 4 the subjects used words such as “columns”, “corridors”, “walls” and “rooms”

to describe the sounds they interacted with. This shows the potential to create

an immaterial architecture made of music that is still spatially navigated. I have

illustrated that the instrumental experience of the participants heavily influenced

their audio-spatial ability; however the architecture itself now has the potential to

become the instrument that musicians and non-musicians alike will inadvertently

play (and thus practice) with their bodies as well as their eyes. This implies that even

non musicians could become musicians of space as they are exposed to this type of

architecture.

Discussion.

30

30

The experiments have not only shown that we perceive spatial inferences in sounds

but also that music can have an affective role in our spatial interaction when the two

are coupled. This affective quality of music could extend the idea of an architecture

made of sound to become active navigational sound architecture for the blind,

where sounds are naturally imbued with spatial characteristics. Ascending scales

of notes could have clear spatial connotations to obstacles such as stairs, or

volumes of sounds with different textures could be used to generate perceptions

of proximity to various objects in a room. The refined musical ability of temporal

pattern recognition could be attuned to the motions of the body as it moves through

space. Phrases of music can now actually become “musical places” which develop

and change as we move from one to another. This combines both uses of the

hippocampus discussed and could be used as a framework to distinguish different

spaces and their relationships to each other. In the same way as Minsky described

musical learning, this system of navigable music would perceptually build into

intuitive wayfinding devices that cue memories of what lies ahead as inhabitants

move through them. These are just a few examples of embodied music concepts

that could contribute to a much larger vocabulary and temporal grammar of intuitive

architectural sound elements.

The implementation and curation of these elements need not be functional and

could become new kinds of architectural ‘detail’. Experiment 4 illustrated how our

visual perceptions of space are overridden by the audible characteristics, in the

same way that film soundtracks significantly enhance or alter the visual narrative.

Pockets of music can now be placed in buildings that change and augment the

qualities of that particular space. The study of how and what music makes us feel is

a larger avenue of research but the architect has the opportunity to change the way

their buildings are listened to as well as seen. The music and the architecture have

the potential to be composed together: Michael Gondry’s film for “Star Guitar” by

the Chemical Brothers is a fantastic example of how music changes our perception

of images, and also how features of space could be heard and augmented in

this way (StarGuitarExperiment.mp4). Every element in the landscape of the train

journey is highlighted by a different sound; attention is drawn to every element in

the scene as it is revealed and audibly signified in the music. Architecture could

use similar technology to literally become audible music in this way, or alternatively

(and perhaps more interestingly) sequences of spaces could be designed with

these principles of Music Aided Design to actively stimulate the same temporal

continuities, contrasts and empathic motions that move us so deeply in music.

31

There is the opportunity to apply this musical intelligence during the appraisal

and brief stages of architectural design. During the contextual analysis of a

site, particular attention could be given to the details, rhythms and forms of the

surrounding spaces and buildings, analysing the developments and changes of

a dynamic perspective through the area. Musical composition could be used to

understand how these rhythmic and dynamic textural changes lead to a dynamic

architectural language of the area as well as just a static, stylistic or materialistic

language. The gap (or silence) of the undeveloped site in the routes through the

area could be conceptually developed based on the expectations, contrasts and

rhythms (to name a few) of the temporal moments either side. This could even be

achieved by composing the music of the context and experimenting with potential

surprises or continuities in the currently empty site. The approaches and departures

from the building could therefore be actively engaging the visual and spatial aspects

of the site with a larger musical narrative. This would continue inside the building

where languages of ‘exposition’, ‘development’ and ‘recapitulation’ (taking sonata

form for example) could be used to create gradual changes, moments of surprise,

steady continuities or moments to reconfirm or change our perceptions of the whole.

The results of this discussion also have implications in other disciplines. Similar

technologies and bespoke software could allow sounds to be actively sculpted,

stretched, smoothed, textured and manipulated as spatial objects. The sounds

could even be passed between multiple collaborators or moulded and developed

together. This could have huge potential in creating music through dance, and also

in electronic sound design and musical composition.

The precedent studies, neurological studies and experimental studies have all

revealed that they may be explored in far greater depth than I have done here.

The experiments deserve a greater sample size to critically examine the changes

between refined spatial awareness and musical ability. The study would also benefit

from exploring the spatial responses to many different types of music and sound

in each experiment. To extend this research, empathy and empathic involvement

with space, motion and music will also need more comprehensive analysis. This

could even extend into how mirror neurons activate the sensorimotor system just by

perceiving the sounds and motions of other objects and people. This dissertation

has been a proof of concept in what clearly has the potential to be a much larger

practical and theoretical pursuit in the disciplines of both architecture and music.

32

32 33

Illustrations & Tables.

Fig. 1: Excerpt from Xenakis’ Metastasis graphical score.

Fig. 2: Excerpt from Xenakis’ Mycenes Alpha graphical score.

Fig. 3: Hyperbolic Paraboloid, Straight lines creating curved forms, diagram.

Fig. 4: Xenakis and Le Corbusier’s Philips Pavilion, photograph.

Fig. 5: Excerpt from Mark Applebaums Metaphysics of Notation graphical score.

Fig. 6: Kiki and Bouba diagrams.

Fig. 7: Superimposed frames of Oskar Fischinger’s Studie Nr. 7 showing expectation and resolution.

Fig. 8: Mark Applebaum playing his bespoke Mouseketeer instrument, photograph.

Fig. 9: Aphasia Echo gesture, film stills with illustrations.

Fig. 10: Aphasia Stretch gesture, film stills with illustrations.

Fig. 11: Guide Psychogeographique de Paris book cover, illustrates relative space mapping, photograph.

Fig. 12: Experiment Setup. Photograph with labels.

Fig. 13: Sketch plan of room and speaker layout, drawing.

Fig. 14: “High” and “Low” frequency sound plane, Illustrated film stills.

Fig. 15: Reaching into Sound Compass, Illustrated film frame - SonicCompass.mp4.

Fig. 16: Searching and finding sounds in space, film stills - SoundsInSpace.mp4.

Fig. 17: Searching and finding sounds in space, Illustrated film stills

Fig. 18: Visualisation of different audible music zones within the space, illustrated photograph.

Fig. 19: Participants correct zone diagram, drawing.

Fig. 20: Participants incorrect zone diagram, drawing.

Fig. 21: Walking backwards and forwards through the music, illustrated film stills.

Fig. 22: Screenshot of motion camera and “Haptic Music” program, screenshot.

Fig. 23: Collecting and releasing music, film stills - HapticMusic1.mp4.

Fig. 24: Crouching to collect the music, film stills.

Fig. 25: Compressing to hold the music, film stills.

Fig. 26: Tangible music, Illustrated photograph.

Table 1: Experiments overview.

Table 2: Experiment numerical data.

Arnold, R (2012). “Theres No Sound In My Head”, Online documentary about the

performances of Mark Applebaum’s “Metaphysics of Notation” https://vimeo.com/14469188

Bengtsson, I & Eggebrecht, H. (1973). Verstehen: Prolegomena zu einem semiotisch-

hermeneutischen Ansatz. In P. Faltin & H.-P. Reinecke (Eds.), Music und Verstehen:

Aufsatze zur semiotischen Theorie, Asthetik und Soziologie der musikalischen Rezeption.

Cologne: Volk.

Borries, F (2007). Space Time Play: Computer games architecture and urbanism, Germany:

Birkhauser

Brown S (1999). The “Musilanguage” Model of Music Evolution. In Wallin NL, Merker B, and

Brown S, eds. The Origins of Music. Cambridge MA: MIT Press.

Clark, A (2008). Supersizing the mind, embodiment, action and cognitive extension, Oxford:

oxford university press

Cupchik, G. C., Phillips, K., & Hill, D. S. (2001). Shared processes in spatial rotation and

musical permutation. Brain and cognition, 46(3), 373-382.

Debord, G. (1957). Guide psychogéographique de Paris. Édité par le Bauhaus imaginiste.

Douglas, K.M., and Bilkey, D.K. (2007). Amusia is associated with deficits in spatial

processing. Nat. Neurosci. 10, 915–921

Dourish, P (2004). Where the Action Is, the foundations of embodied interaction, Cambridge

MA: MIT Press.

Bibliography.

34

Ekstrom A.D, Kahana M.J, Caplan J.B, Fields T.A, Isham E.A, Newman E.L, Fried I. (2003).

Cellular networks underlying human spatial navigation. Nature 425:184–188.

Erneling, C. E., & Johnson, D. M. (Eds.). (2005). The mind as a scientific object: between

brain and culture. Oxford University Press: USA.

Fishwick, P. A, (2006). Aesthetic Computing, Cambridge MA: MIT Press

Gaser, C., & Schlaug, G. (2003). Brain structures differ between musicians and non-

musicians. The Journal of Neuroscience, 23(27), 9240-9245.

Gray J.A, Rawlins J.N.P. (1986). The hippocampus, Comparator and buffer memory: an

attempt to integrate two models of hippocampal function. Plenum: New York, p159–201.

Hatten, R. S. (1994). Musical meaning in Beethoven: Markedness, Correlation, and

Interpretation. Bloomington: Indiana University Press

Hebb, D. O. (1949). The Organisation of behaviour: A Neuropsychological theory. New York:

John Wiley and sons.

Herdener, M., Esposito, F., di Salle, F., Boller, C., Hilti, C. C., Habermeyer, B., ... & Cattapan-

Ludewig, K. (2010). Musical training induces functional plasticity in human hippocampus.

The Journal of Neuroscience, 30(4), 1377-1384.

Hofmann, B (2005). Spatial Aspects in Xenakis’ Instrumental Works. In Makis Solomos,

Anastasia Georgaki, Giorgos Zervos (ed.), Definitive Proceedings of the “International

Symposium Iannis Xenakis” (Athens, May 2005), www.iannis-xenakis.org, October 2006

Howard, D (2009). Sound and Space in renaissance venice, China: Yale University Press

Huron, D. (2006). Sweet Anticipation: Music and the Psychology of Expectation. Cambridge,

MA: MIT Press.

Husserl, E, (1928). Lectures on the Phenomenology of Internal Time Consciousness,

Bloomington : Indiana University Press

Köhler, W (1929). Gestalt Psychology. New York: Liveright.

34

Knight, R (1996). Contribution of human hippocampal region to novelty detection. Nature

383:256–259.

Knights S, (2012). Visual Music: the films of Oskar Fischinger. Cambridge: History of Art

dept. Dissertation

Kumaran D, Maguire EA (2007). Which computational mechanisms operate in the

hippocampus during novelty detection? Hippocampus 17:735–748.

Lakoff, G., & Johnson, M. (1980). Metaphors we live by (Vol. 111). London: Chicago.

Lakoff, G. J., & Johnson, M. M.(1999). Philosophy in the Flesh. The Embodied Mind and Its

Challenge to Western Thought. New York: Basic Books.

Leman, M, (2008). Embodied Music Cognition and Mediation Technology. Cambridge MA:

MIT Press

Lidji, P., Kolinsky, R., Lochy, A., & Morais, J. (2007). Spatial associations for musical

stimuli: A piano in the head?. Journal of Experimental Psychology: Human Perception and

Performance, 33(5), 1189.

Maguire, E.A. (2001). Neuroimaging, memory and the human hippocampus. Rev Neurol

Paris 157:791–794.

Maguire, E.A. Gadian D.G, Johnsrude I.S, Good C.D, Ashburner J, Frackowiak R.S, Frith C.D.

(2000). Navigation-related structural change in the hippocampi of taxi drivers. Proc Natl

Acad Sci USA 97: 4398-4403.

Mallgrave, H. F, (2010). The Architects Brain. UK: Wiley-Blackwell

Maurer D, Pathman T & Mondloch CJ (2006). The shape of boubas: Sound-shape

correspondences in toddlers and adults. Developmental Science 9 (3): 316–322

Minsky, M (1981). Music, Mind and Meaning. In Clynes M, ed. Music, Mind and the Brain:

The Neuropsychology of Music. Plenum: New York

Piaget, J. (1970). Piaget’s theory.

35

Pratt, C.C (1930). The spatial character of high and low tones Journal of Experimental

Psychology. 13 (1930), pp. 278–285

Merleau-Ponty, M. (1996). Phenomenology of perception. Motilal Banarsidass Publishe.

Ramachandran, V. S., & Hubbard, E. M. (2001). Synaesthesia - a window into perception,

thought and language. Journal of Consciousness Studies, 8(12), 3-34.

Rasmussen, S. E. (1964). Experiencing architecture (Vol. 2). Mit Press.

Rodrigues, A. C., Loureiro, M. A., & Caramelli, P. (2010). Musical training, neuroplasticity and

cognition. Dementia & Neuropsychologia, 4(4), 277-286.

Rusconi E, Kwan B, Giordano B.L, Umilta C, Butterworth B (2006). Spatial representation of

pitch height: The SMARC effect. Cognition, 99 (2), pp. 113–129

Sacks, O, (2008). Musicophilia; Tales of music and the Brain, New York: Knopf

Shapiro, L, (2010). Embodied Cognition. USA: Routledge

Stewart, L., & Walsh, V. (2007). Music perception: sounds lost in space. Current Biology,

17(20), R892-R893.

Strange B.A, Fletcher P.C, Henson R.N, Friston K.J, Dolan R.J. (1999). Segregating the

functions of human hippocampus. Proc Natl Acad Sci U S A 96:4034–4039.

Strange B.A, Dolan R.J. (2001). Adaptive anterior hippocampal responses to oddball stimuli.

Hippocampus 11:690–698.

Tversky, B. (2008). Spatial cognition: Embodied and situated. The Cambridge handbook of

situated cognition, 201-217. USA: Cambridge University Press.

Van Gelder, T, (1998). The dynamical hypothesis in cognitive science, Behav. Brain Sci. 21,

615–628. doi: 10.1017/S0140525X98521735.

Vinogradova, O.S. (2001). Hippocampus as comparator: role of the two input and two output

systems of the hippocampus in selection and registration of information. Hippocampus

11:578–598.

Appendix:

36

(Volunteer data is recorded in hardcopy)

www.felixfaire.com