MA R IA MA N N O N E

Networks of Music and Images

Introduction

hile hearing a complex orchestral piece, have you ever experienced “seeing” a

fascinating landscape in your mind? Might this have something to do with

synesthesia? Synesthesia is a sensory stimulation that evokes feelings in both

the initial and at least one additional sensory channel. “Synesthesia” comes from ancient

Greek,1 and is used as a concept in poetry («L’urlo nero», the «black scream», for example).2

This topic is also discussed in the fields of psychology, neurology, and, of course, studies

about arts. Sonification experiments may benefit from studies about synesthesia, and more

generally, psychology, to find which sonification technique may be most effective for

listeners. There is a topic of study akin to synesthesia: cross-modal correspondences. While

synesthesia is more or less an involuntary experience, cross-modal associations are thought

to be more organized, structured, and statistically-based experiences.3 Synesthesia is often

considered as a specific neurological condition, while cross-modal correspondences may be

more largely (and maybe culturally) shared. Research showed that also non-synesthetic

people can experience cross-modal correspondences.4 Here we will not deal with details

about their similarities and differences. We will only focus on the broad concept of superpo-

sition between simultaneous and different sensory-channel experiences (i.e. the association

between stimuli from different sensory fields).

W

Scholars of synesthesia and cross-modal correspondences develop the analysis of

similar information from different fields – sounds, visuals, kinetics. Sometimes, they frame

1 It comes from σύν “together” and αἴσθησις “sensation”.2 «[…] All’urlo nero della madre / che andava incontro al figlio / crocifisso sul palo del telegrafo» from the

poem Alle fronde dei salici within Giorno dopo giorno (1947) written by Salvatore Quasimodo, Nobel prize forthe Literature in 1959.

3 Cf. CHARLES SPENCE, Crossmodal correspondences: A tutorial review, «Attention, Perception, & Psychophysics»,LXXIII, 4 (2011), pp. 971-995; OPHELIA DEROY - CHARLES SPENCE, Why we are not all synesthete (not even weakly so) ,«Psychonomic Bulletin & Review», XX, 4 (2013), pp. 643-664.

4 Cf. CESARE PARISE - CHARLES SPENCE, Audiovisual crossmodal correspondences and sound symbolism: a study using theimplicit association test, «Experimental Brain Research», CCXX, 34 (2012).

Gli spazi della musica vol. 6, n. 2 (2017)ISSN: 2240-7944

http://www.ojs.unito.it/index.php/spazidellamusica

MARIA MANNONE

their studies in the context of Gestalt, especially while dealing with the perceptual

grouping5 and music.6 Other studies deal with the perception of sound and the localization

of points in space. A higher-frequency sound is more likely to be localized at a higher point

in the space. In contrast, a low-frequency sound is more likely to be localized at a lower

point in the space.7 There are several studies in the field.8 There are also several studies

connecting the spectrum of speech and the sharpness/smoothness of the visuals’ shape. An

example of this is the association between (meaningless) words and lines, an object of

experimental research in psychology. Words and lines can be associated depending on their

degree of “angle-ness” or smoothness, as shown in a well-known experiment based on

Gestalt principles (the “takete-maluma” experiment).9 Recently, the results of this exper-

iment have been analyzed in the light of psychophysics,10 associating “angle-ness” in speech

with explosive sounds, increased frequency, and shorter duration.11 Analogies between

images and sound (“angle-ness” and “explosivity”) interpreted in terms of gestures is also a

theme addressed by scholars,12 and is the topic of iconicity13 studies.14 The binding rela-

tionship between visual shapes and gestures has been investigated in the formation of

5 Cf. LUCA NOBILE, Words in the mirror: analysing the sensorimotor interface between phonetics and semantics in Italian,in Iconicity in Language and Literature 10: Semblance and Signification , ed. by Pascal Michelucci - Olga Fischer -Christina Ljungberg, Amsterdam – Philadelphia, John Benjamins, pp. 101-131; SUZANNE ROFFLER - ROBERT BUTLER,Factors that influence the localization of sound in the vertical plane, «The Journal of the Acoustical Society ofAmerica», XLIII (1968), pp. 1255-1259; MICHAEL KUBOVY - MINHONG YU, Multistability, cross-modal binding and theadditivity of conjoined grouping principles, «Philosophical Transactions of the Royal Society B», CCCLXVII(2012), pp. 954-964.

6 Cf. FRED LERDAHL - RAY JACKENDOFF, A Generative Theory of Tonal Music, Cambridge (MS), MIT, 1983.7 Cf. S. ROFFLER - R. BUTLER, Factors that influence the localization of sound in the vertical plane, cit.8 Cf. ELENA RUSCONI [et al.], Spatial representation of pitch height: the SMARC effect, «Cognition», XCIX, 2 (2006),

pp. 113-129.9 Cf. WOLFGANG KÖHLER, Gestalt psychology, New York, Liveright, 1929; DIMITRI UZNADZE, Ein experimenteller Beitrag

zum Problem der psychologischen Grundlagen der Namengebung, «Psychologische Forschung», V, 1-2 (1924),pp. 24-43; M. KUBOVY - M. YU, Multistability, cross-modal binding and the additivity of conjoined grouping principles,cit.

10 Correspondences between the perception of sounds and the perception of images are the object of recentexperimental research: cf. C. PARISE - C. SPENCE, Audiovisual crossmodal correspondences and sound symbolism: astudy using the implicit association test, cit. Crossmodal correspondences are also studied in the field ofneuroscience, cf. MAURIZIO GENTILUCCI - PAOLO BERNARDIS, Imitation during phoneme production, «Neuropsychologia»,VLV, 3 (2007), pp. 608-615.

11 Cf. L. NOBILE, Words in the mirror: analysing the sensorimotor interface between phonetics and semantics inItalian, cit.

12 Cf. LAWRENCE ZBIKOWSKI, Conceptualizing Music: Cognitive Structure, Theory, and Analysis, Oxford, Oxford UniversityPress, 2002.

13 Iconicity is the relationship between the shape of a sign and its meaning.14 Cf. WILLI MAYERTHALER, System-independent morphological naturalness, in Leitmotifs in Natural Morphology, ed. by

Wolfgang U. Dressler [et al.], Amsterdam – Philadelphia, John Benjamins Publishing, 1987, X, pp. 25-58.; cf.ADRIEN MERER [et al.], Perceptual Characterization of Motion Evoked by Sounds for Synthesis Control Purposes , «ACMTransactions on Applied Perception (TAP)», X, 1 (2013).

– 39 –

Networks of Music and Images

audio-visual objects.15 Later, we will give more details about the connection between visual

shapes and sounds via gestures.

Here, we will consider examples from the artistic practices of drawing and playing

(acoustic) musical instruments. In particular, we discuss pitch in music and space in visuals,

the two “indispensable attributes” for auditory and visual channels, respectively.16 There is

another concept from Greek thinking we can use: Plato’s abstraction concept, the idea.17

According to Plato, all that surrounds us is a copy of perfect things, the ideas. For example,

all existing horses are copies of the “horse” idea. In this article, we need this concept to

explain abstraction and universal forms, to be used as unifying elements of networks.

To connect sound and visuals, we refer to a specific technique of tridimensional

mapping.18 The coordinates (length, width, height) of a (finite) selection of points of a tridi-

mensional object can be associated with musical parameters such as time, loudness, and

pitch, see Figure 1.

15 Cf. M. KUBOVY - M. SCHUTZ, Audio-Visual Objects, «Review of Philosophy and Psychology», 1 (2010), pp. 41-61.16 Ibidem.17 PLATO, The collected dialogues, ed. by Edith Hamilton - Huntington Cairns, New York, Pantheon, 1961.18 Cf. MARIA MANNONE, Dalla Musica all’Immagine, dall’Immagine alla Musica. Relazioni matematiche fra composizione

musicale e arte figurativa, Palermo, Edizioni Compostampa, 2011; GUERINO MAZZOLA - MARIA MANNONE - YAN PANG,Cool Math for Hot Music, Heidelberg, Springer, 2016.

– 40 –

Fig. 1: Duchamp’s Bicycle Wheel within thetridimensional space time-loudness-pitch. Therepresentation is used to map the points of theimage into sounds. It has been built to composean orchestral score. Drawing by Maria Mannone.

MARIA MANNONE

In this way, we can map visual shapes into sound. Timbre is a free parameter, but an

additional mapping of parameters, for example from visual color to musical timbre,19 can be

defined and applied. Such a technique may be seen as the development of Xenakis’ UPIC20

and studies about the visual representation of scores.21

Our conjecture is the following: effective examples of such a technique can satisfy

cross-modal correspondences. For example, we can musically re-create the shape of Gaudí’s

Sagrada Família with raising and lowering musical scales, or the Concetto Spaziale by Lucio

Fontana via acciaccature and staccato articulations.22 We can envisage the same gestural

matrix for both: a “staccato” movement (i.e. one that is short and precise) may generate

points (and even holes) on canvas just as it may create detached notes on musical instru-

ments.23

As shown in literature,24 the auditory channel is specialized in detecting “where” is

the sound source, and the visual channel in getting information about “what” surfaces in

space.

Let us suppose that the “where” is changing, and the sound source is moving in

space. We can decide to freely use the concept of sound localization and surface description,

as well as another concept, the figure-ground segregation applied to sound, to describe a

surface via a moving sound source. We can apply these ideas to musically render a visual

shape such as a water lily. We can use two simultaneous melodies, one raising and the other

lowering, for each petal. To represent the conic shape of the water lily, we can imagine

ourselves “walking along” the petals, hearing a couple of melodies for each petal, and a

petal after the other. To make the feeling of “movement” more evident, we can use a

different timbre for each petal, and slightly change the pitch and the loudness, mimicking

the Doppler effect in physics. The Doppler effect describes the frequency change that is

perceived by the listener when he or she is moving with respect to the sound source (the

19 We are comparing color and timbre not as two ends of a spectrum, but as a semi-expansive list of possibleapplications.

20 Cf. IANNIS XENAKIS, Formalized Music, Stuyvesant (NY), Pendragon Edition, 1971.21 Cf. SALVATORE SCIARRINO, Le figure della musica: da Beethoven a oggi, Milano, Ricordi, 1998.22 Cf. M. MANNONE, Dalla Musica all’Immagine, dall’Immagine alla Musica. Relazioni matematiche fra composizione

musicale e arte figurativa, cit.23 The basic mapping is: length-width-height into time-loudness-pitch, and we add the condition of gestural

similarity (see Section 1). This is a creative choice. There is not explicitly evidence that such a mappingalone fulfills embodied cognition requirements. Further research will involve cognitive experiments to testour conjecture.

24 Cf. M. KUBOVY - M. SCHUTZ, Audio-Visual Objects, cit.

– 41 –

Networks of Music and Images

frequency shift is not produced by the sound source). Thus, we can musically reproduce the

illusion of movement through the perception of frequency change. For example, we may

hear higher and louder sounds when we move toward a petal, and lower and softer sounds

when we move away from a petal. Such a compositional experiment is described in a recent

dissertation, which includes an orchestral score and its reading.25 The connection between

music, motion, and the perception of motion as an “image” created in the mind of listeners,

is also a topic of experimental research.26

Conveniently, a branch of math known as category theory is particularly suitable in

describing objects and transformations between them.27 An object, represented by a point,

can be transformed into another object (represented by another point). The transformation

(morphism) is represented by an arrow. Arrows can be mathematically composed, and their

associativity can be studied. A category is given by a set of objects connected by arrows,

satisfying associative and identity properties. In category theory we can build complex,

nested structures starting with these simple ideas. In our frame, we can see a musical

sequence as an object, and the variation process as an arrow. The same can be applied to

visual arts. From a more broad perspective, we can argue that, due to the developments of

modern mathematics, a convergence of disciplines may be envisaged. This fact is remi-

niscent of the interdisciplinary approach to knowledge in Renaissance culture.

In summary, we can transform musical objects into other musical objects, visual

shapes into musical shapes, and we can map sounds into images and vice versa.

Let us take this one step further. Once we define a sequence of transformations of

images, we can map each one onto music, creating a parallel sequence of different images.

In principle, if we have a network of images, we can also have an associated network of

musical sequences, connected through mappings, such as length-width-height to time-

loudness-pitch. The images in the network will have in common a similar structure

(something we might compare with the Platonic “idea”), as well as the network of musical

sequences. An example of a visual network, based upon the idea of “triangle” and investi-

gated through computer sciences within the frame of category theory, is given in litera-

25 Cf. MARIA MANNONE, Musical Gestures between Scores and Acoustics: A Creative Application to Orchestra, Ph.D. disser-tation, University of Minnesota, 2017. Recording: https://soundcloud.com/maria-mannone/mannone-genesis-of-music-from-gestures-second-part-fugue, from 5’05’’ to 6’20’’.

26 Cf. ZOHAR EITAN - RONI GRANOT, How Music Moves: Musical Parameters and Listeners’ Images of Motion, «MusicPerception», XXIII, 3 (2006), pp. 221-247.

27 Cf. SAUNDERS MAC LANE, Categories for the Working Mathematician, Heidelberg, Springer, 1971.

– 42 –

MARIA MANNONE

ture.28 Where it is also stated that “the network is the abstraction”. The network is given by

a collection of images with a common characteristic (that in literature29 is the triangular

shape), connected by arrows. The arrows represent the transformation that brings an image

into another. The final arrow leads to an image reminding to “perfect triangle” (that is a

pure idea, and cannot even be drawn as a specific triangle), the colimit of the entire

collection of triangles. In building the network of visuals, the authors used concepts and

terminology from category theory.30 In our article, we extend these ideas to music, and to

image-music interactions. In this way, we may ideally connect concepts from ancient

philosophy and abstract mathematics to contemporary artistic and computer applications.

The structure of the article is the following. In Section 1, we describe networks of music and

images using the concept of gestural analogy and the frame of category theory. In Section 2,

we describe some examples of networks, using the example of music derived from the shape

of cathedrals. Finally, we comment on the results and discuss further research in the

Conclusion Section.

Musical Gestures, Diagrams, and Networks

In this Section, we join concepts of synesthesia (in a broad sense), (musical) gesture,

category, diagrams, and visuals to start building networks of music and image, giving

instruction for a creative technique. We can combine synesthesia and networks via a simple

syllogism: 1. music can be described by category theory;31 2. category theory can be success-

fully applied to visuals;32 thus, 3. also interactions and translations between music and

images can be framed within category theory.33 Because networks in literature34 use cate-

28 Cf. LEONIDAS GUIBAS, Networks of Shapes and Images, Lecture ICVSS, 2014, http://www.slideshare.net/milkers/lecture-07-leonidas-guibas-networks-of-shapes-and-images (slides 122-123), and SHUBHAM TULSIANI [et al.], Learning ShapeAbstractions by Assembling Volumetric Primitives, ArXiv, 2016, https://arxiv.org/pdf/1612.00404v1.pdf.

29 Cf. L. GUIBAS, Networks of Shapes and Images, cit.30 The concept of categorization, in a more broad sense than strictly mathematical, as well as classification, is

also present in other works about perception: cf. for example JOHN PAUL MINDA - DAVID SMITH, Prototypes in cate-gory learning: the effects of category size, category structure, and stimulus complexity, «Journal of ExperimentalPsychology: Learning, Memory, and Cognition», XXVII, 3 (2001), pp. 775-799, and ROBERT NOSOFSKY, Tests of anExemplar Model for Relating Perceptual Classification and Recognition Memory , «Journal of Experimental Psycho-logy», XVII, 1 (1991), pp. 3-27. Experiments about similarity in perception may profit from mathematicalframing, especially regarding a predictive model to test, and the analysis of results to improve the theore-tical model.

31 Cf. S. MAC LANE, Categories for the Working Mathematician, cit; DAVID SPIVAK, Category Theory for the Sciences,Cambridge (MS), MIT, 2014; G. MAZZOLA - M. MANNONE - Y. PANG, Cool Math for Hot Music, cit.

32 Cf. L. GUIBAS, Networks of Shapes and Images, cit.33 Cf. MARIA MANNONE, Introduction to Gestural Similarity in Music. An application of Category Theory to the Orchestra ,

«Journal of Mathematics and Music», submitted 2017.34 Cf. L. GUIBAS, Networks of Shapes and Images, cit.

– 43 –

Networks of Music and Images

gories, we propose here a joint use of networks and categories for sounds and visuals. Such

a syllogism is coherent according to category theory. We may conjecture that, if each step

of the process verifies cross-modal correspondences, also the entire networks verify them,

and if we move from a network to another, we can be able to listen to shapes and to see

sounds.

We need one more concept, the concept of musical gestures. There is an extended

research on musical gestures.35 The perspective on music and embodied cognition integrate

the mathematical definition of gesture, that is meant as a formalization of such an embod-

iment. In the mathematical theory of music, a gesture is defined as a mapping of an abstract

structure of points and arrows, onto a system of continuous curves in space and time

(trajectory in a generalized space),36 see Figure 2 for a simple example. Such a basic and

abstract definition can be applied to a variety of cases, from piano movements to violin,

conducting, and even to voice (if we consider the inner movements made by a singer). 37

Focusing on this system of continuous curves, we can compare gestures on the same

musical instrument, and between different musical instruments. Because the comparisons

between gestures are reduced in this way to a comparison between curves, we can use

mathematical tools of homotopy theory to classify musical gestures. Such a tool should be

joined with the analysis of the musical spectra obtained as results of musical gestures for a

more complete description.

35 Cf. Musical Gestures: Sound, Movement, and Meaning, ed. by Rolf Inge Godøy - Marc Leman, New York –Abingdon, Routledge, 2010.

36 Cf. GUERINO MAZZOLA - MORENO ANDREATTA, Diagrams, Gestures, and Formulae in Music, «Journal of Mathematics andMusic», I, 1 (2007), pp. 23-46.

37 Cf. M. MANNONE, Introduction to Gestural Similarity in Music. An Application of Category Theory to the Orchestra, cit.;G. MAZZOLA - M. MANNONE - Y. PANG, Cool Math for Hot Music, cit.

– 44 –

Fig. 2: Mathematical definition of a gesture as a mapping from askeleton (on the left: an abstract structure with points and arrowsconnecting them) to a body (on the right: a system of continuouscurves in a space, that can be, for example, x = time, y = verticalposition of the hand, and z = horizontal position of the hand).

MARIA MANNONE

In literature,38 two musical gestures are called “similar” if their curves in space and

time can be homotopically transformed one into the other, and if their sound spectra show

consistent analogies (e.g. they both correspond to vibrato or non vibrato).39 If we consider

drawing as the result of a specific gesture, or, in general, a visual artwork as the result of a

painting/sculpting gesture, we can easily generalize the concept of gestural similarity

beyond music. Comparisons between music and visuals are easy in this way. The same

detached gesture can generate a staccato musical fragment, as well as a collection of points

on the canvas. The same happens with a more continuous movement, and so on. In the same

article40 orchestral conductors’ movements, using the terminology of category theory, are

described as the colimit of all the orchestral musicians’ gestures, and the listeners’

projections as the limit of all sounds. In the case of feedback (e.g. the orchestra influencing

the conductor’s gestures, or the public influencing the musicians’ performance) the arrows

can just be reversed.

Now, let us revisit topic of mappings and networks. We can define networks of

gestures with progressive ramifications and transformations. Because, as said in the Intro-

duction, the same gesture can generate either an image or a sound on acoustic instruments,

we can see networks of gestures as the connection between networks of visuals and

networks of (mapping-related) sounds. In fact, we can see an image as the result of a

drawing gesture.41 See Figure 3 for a simple example of gesture, image, and sound networks

and diagrams. In Figure 3, we have arrows from Image to Sound, and from Gesture to Sound.

The inverse arrows, i.e., from Gesture to Sound and from Sound to Image, can represent the

inverse transformations. Figure 3, in fact, is only an example of how a network may be built,

from Gesture 1, Image 1, Sound 1, to Gesture 2, Image 2, Sound 2 and so on. The diagram

does not represent the most general case. The path Sound 1 to Sound 2 to Sound 3 is a

fragment of an audio network. The more general path is Image to/from Gesture to/from

Sound. The choice of a specific direction allows us to study precise commutative diagrams 42

one after the other.

38 Cf. M. MANNONE, Introduction to Gestural Similarity in Music. An application of Category Theory to the Orchestra, cit.39 The measure of similarity is continuous: e.g., we can use a decimal within the interval [0, 1] (0 for minimal

similarity and 1 for maximal similarity).40 Cf. M. MANNONE, Introduction to Gestural Similarity in Music. An application of Category Theory to the Orchestra, cit.41 Cf. A. MERER [et al.], Perceptual Characterization of Motion Evoked by Sounds for Synthesis Control Purposes , cit.; M.

MANNONE, Introduction to Gestural Similarity in Music. An application of Category Theory to the Orchestra, cit.42 In category theory, a commutative diagram is a diagram where arrows (representing paths) can be

composed, and the composition of different arrows with the same start and end points lead to the sameresult.

– 45 –

Networks of Music and Images

In Guibas’ work43 the concept of the Platonic idea for “cow” is seen as the final

element of a collection of cows, and a simplification of the cow’s skeleton structure as its

colimit. If we see such a structure as the result of a drawing gesture in the tridimensional

space, we have successfully linked gestures to the Platonic idea.

We now propose a method to build networks of sounds and visuals. This can be used

merely as a compositional technique, but it can also be a tool for programming, as well as a

tool to analyze and compare artworks, and (hopefully) to deepen our knowledge of synes-

thetic processes.

Given an image or a collection of images, we can:

modify the images, then convert each of them into music, and finally, listen to the

different sounds;

modify a musical fragment by changing its articulation, timbre, pitch, or other

parameters, then convert the result into a visual shape;

compare the new image with the one derived from the initial musical fragment;

create a separate network for each image and sound, and compare the results at

each separate step for both networks.

All that we have heretofore covered, may be applied on a computer software for

educational, research, and creative purposes. As implied by Guibas,44 visual arts can profit

from diagrammatic techniques of visual rendering and shaping (techniques described by

Guibas45 do not deal with sound-to-image conversion, but with the definition of visual

43 Cf. L. GUIBAS, Networks of Shapes and Images, cit.44 Ibidem.45 Ibidem.

– 46 –

Fig. 3: An example of a Gesture- Image - Sound network.

MARIA MANNONE

networks and categorical concepts). We can say that even interactions between music and

visual arts46 can be improved via diagrammatic thinking.47 We already use software and

programming languages to transform music, such as Max/MSP, Open Music (based on Lisp),

and Rubato Composer, with the application of gestural interaction. Technology such as 3d-

image manipulation48 can be applied to create musical material via the tridimensional

mapping described above.49 The resulting sounds represent a starting point for the sound

art, for musical informatics purposes, and they can be transcribed and played on acoustical

instruments. Examples of networks of music and images are given in Section 2. In fact, we

can design and develop software that automatically composes music from the gestural and

visual analysis. Programming applications may help cognitive experiments, as asking the

research question: Is this musical rendition effective?

A case study: Music of Cathedrals

We present a simple case study to show the proposed methodology. We will give

some examples of what music-image networks are, and how to build them. This is a simple

compositional example to give an idea, but the applicability of such a method goes far

beyond this, including, in principle, sets of experiments about perception. The networks can

also be defined depending on the analogies experimentally found between sequences of

images, sequences of sounds, and their connections. Here, the chosen idea is the shape of

some cathedrals. The profile of each cathedral is indicated in red in Figures 4, 5, and 7.

Figure 4 shows an example of point (a) in Section 1: the modifications in the image

are converted into music. We have three different cathedrals, and we envisaged a little

network where the three images are ideally transformed, one into the other, through a

progressive addition of spires. We composed the three short musical sequences starting

from the highlighted shape of each cathedral. In a nutshell, vertical lines are associated to

chords and clusters of notes, and oblique lines as ascending and descending scales. We

rendered the horizontal lines with the repetition of the same note. We constructed each

musical sequence separately from each visual shape, but the three sequences can be seen as

part of another little network, with three elements, transformed one into the other,

increasing the musical variety. In this way, we built two parallel networks: one for visual

46 Cf. M. MANNONE, Introduction to Gestural Similarity in Music. An application of Category Theory to the Orchestra, cit.47 Cf. CHARLES ALUNNI, Diagrammes et Catégories comme prolégomènes à la question: Qu’est-ce que s’orienter

diagrammatiquement dans la pensée?, «Théorie Littérature Epistémologie», XXII (2004), pp. 83-93.48 Cf. L. GUIBAS, Networks of Shapes and Images, cit.49 Cf. M. MANNONE, Introduction to Gestural Similarity in Music. An application of Category Theory to the Orchestra, cit.

– 47 –

Networks of Music and Images

shape, the other for music, both with increasing complexity. We may show people the three

images and the three musical sequences, without their established order, asking to “re-

order” them. We expect that people would choose the order of increasing complexity, or

decreasing complexity. Such an experiment may validate the construction of parallel

networks proposed here, and similar experiments may constitute the development of our

work.

In Figure 5, a musical sequence composed from a visual shape is progressively

modified (point b). The final musical fragment is converted into an image and compared

with the initial one (point c). Here, we start the musical network from a single visual shape.

After several purely musical variations, the melodic contour is transformed back into a

visual shape, and it is compared with the initial one. In this way, we can construct visual

variations through (hidden) musical variations.

– 48 –

Fig. 4: The profile of a cathedral is ideally transformed into another (red linesindicate the cathedral profiles), constituting a simple visual network. Eachprofile is mapped into an original short musical fragment, and the sequence ofthe three musical sequences constitute a simple sound network connected withthe visual one.

MARIA MANNONE

Finally, Figure 6 demonstrates a network for musical fragments. The “abstract idea”

is here a very simple structure with a cluster, a melodic cell, and again another cluster. In

Figure 7, we have a network of visual shapes of cathedrals in different styles and places.

Also here, the “abstract idea” is a simple vertical-horizontal-vertical line, that is the visual

analogous of the cluster-melody-cluster of Figure 6. The networks of Figure 6 and 7 are

independent; they only have in common this simple “abstract idea”, derived from the

essential shape vertical line—horizontal line—vertical line.

– 49 –

Fig. 5: A short musical fragment is derived from the profile of a cathedral.Then, the fragment is progressively variated, and finally re-transformed intoa visual shape. The final visual shape can be compared with the initial one.

Networks of Music and Images

Translations from visual shapes into music can use algorithms and they can involve

other musical parameters such as the change of timbre and articulation. For sake of simplic-

– 50 –

Fig. 6: An example of musical network, with the “abstract idea” given by a simplesequence chord/cluster—melodic cell—chord/cluster. This is the visual equivalent of thenetwork of Figure 7.

Fig. 7: An example of visual network, with the “abstract idea”given by a simple profile “vertical line—horizontal line—verticalline. This is the visual equivalent of the network of Figure 6.

MARIA MANNONE

ity, the proposed examples only involve pitch and onset, due to the easiness of the

perception of raising/lowering pitch as ascending/descending movement in the tridimen-

sional space.50

There is not only one way to transform visuals into music and vice versa. The chosen

example was meant to explain the structure of visual, musical, and music-visual networks.

As previously described, examples of Figure 1 to 4 can be themselves objects of experi-

mental analysis in the frame of perception.

Summarizing, in this Section we analyzed some examples of visual and musical

networks. We described the technique to make them, not only as a compositional tool, but

also as a method that can be applied in programming and can be used for (and improved

through) perception experiments.

Conclusions

In this article, we joined several ideas: 1. the concept of a network of visuals with

elements of category theory; 2. the structure of category theory applied to music; 3.

mappings image-music.

We ideally connected to music the concepts of synesthesia and Platonic abstraction,

extensively treated in the literature about image networks, and we made reference to

contemporary mathematics to describe and connect music and visual arts. In fact, the

technology developed for visuals can be extended to music exploiting cross-modal corre-

spondences. While connecting the elements 1, 2, and 3, we hypothesized the existence of

networks of musical sequences, and networks of visuals plus music with mutual interaction

and exchanges.

At this stage of the research, our contribution is mainly bound to music composition,

general artistic creativity, and education. However, after a collection of enough data, we

may think of broadening the field of research including possible applications to machine

learning. In fact, the definition of visual and musical networks can enhance the collabo-

ration between scholars of different fields, included artificial intelligence, where algorithms

to compare and classify images are developed.51 For example, machine learning applied to

music may profit from these new concepts by incorporating visual comparisons. Perception

experiments proposed to musicians, visual artists, as well as to non-musicians and non-

50 Cf. S. ROFFLER - R. BUTLER, Factors that influence the localization of sound in the vertical plane, cit.51 Cf. KEVIN KIRBY, On Memory, Neurons, and Rank-One Connections, «The College Mathematics Journal», XXVII

(1997), pp. 2-19.

– 51 –

Networks of Music and Images

artists, can help in refining the technique to develop networks, and give a stronger, experi-

mental base to our hypothesis of connections. There are several studies about the

perception of sound evoking motion, drawing-motion-gestures,52 and the perception of

music depending on gestures;53 we may compare our proposed methodology with their

work. Experiments may also confirm which mappings are more effective than others,

approaching cross-modal correspondence studies, with Gestalt-inspired theory such as

audio-visual object studies,54 to musicology and analysis of visual arts.

All this knowledge may find new applications in the design of human-computer

interfaces, helping to achieve the goal of making them more intuitive and user-friendly.

Software to modify images may include sounds, or vice versa. This may be helpful to

develop specific software for visually-impaired people, or other programs to help hearing-

impaired people. This work may also enhance artistic creativity as well as deepen its theore-

tical understanding of arts. Finally, both visual and auditory network studies can help to

develop new strategies for making more intuitive and accessible abstract concepts from

mathematics, such as category theory and network theory. Further developments should

also include cognitive experiments, starting from a simple listening test, where subjects are

asked to listen/see shapes and draw/sing them (“Sing this Draw” and “Draw this Song”).

Such a test could be run by using the cathedrals profile and the corresponding musical

sequence we composed.

Summarizing, future developments of research about music and visual networks

may involve computer science, human-computer interfaces, theoretical thinking on art and

science interactions, as well as creative applications of scientific ideas. An interdisciplinary

cooperation between artists and scientists can help non-scientists to draw abstract concepts

from mathematics as well as non-artists to approach the complexity and richness of art.

N O TE

About the examples, according to the editorials guidelines the author has verified, underhis own responsibility, that the reproductions are not covered by copyright: otherwise,he obtained from the copyrights holders consent to the publication.

52 Cf. ADRIEN MERER [et al.], Perceptual Characterization of Motion Evoked by Sounds for Synthesis Control Purposes, cit.53 Cf. MICHAEL SCHUTZ - SCOTT LIPSCOMB, Influence of visual information on auditory perception of marimba stroke types , in

Proceedings of the 8th International Conference on Music Perception & Cognition, ed. by Scott Lipscomb [et al.],Evanston (IL), Causal Productions, 2004, pp. 76-80; cf. MICHAEL SCHUTZ - SCOTT LIPSCOMB, Hearing Gestures, SeeingMusic: Vision Influenced Perceived Tone Duration, «Perception», XXXVI, 6 (2007), pp. 888-897.

54 Cf. M. KUBOVY - M. SCHUTZ, Audio-Visual Objects, cit.

– 52 –