Plug-In to Fear - Game Biosensors and Negative Physiological Responses to Music 8.1.Mitchell

7/24/2019 Plug-In to Fear - Game Biosensors and Negative Physiological Responses to Music 8.1.Mitchell

1/22

Music and the Moving Image, Volume 8, Number 1, Spring 2015, pp.

37-57 (Article)

For additional information about this article

Access provided by University of California @ Berkeley (20 Oct 2015 03:39 GMT)

http://muse.jhu.edu/journals/mmi/summary/v008/8.1.mitchell.html
http://muse.jhu.edu/journals/mmi/summary/v008/8.1.mitchell.htmlhttp://muse.jhu.edu/journals/mmi/summary/v008/8.1.mitchell.html


2/22

. /

Introduction

Over recent decades, the games industry hassurged forward in a relentless tide of techno-

logical change. Driven by innovation, inven-tion, and growing market demand, the in-dustry continues to explore new approaches,all in search of the ultimate gaming experi-ence. A number of current developments in-dicate that the industry is keen to explore thepotential of the human body itself, and asso-ciated physiological responses, as the focusof future interface development. In

Sony filed a patent application for biometriccontrols (designed to measure electromus-cular data, electrocardio data, and galvanicskin resistance).Microso has developedplans for a Joule heart-rate monitor for theKinect Play Fit, and Nintendo experimentedwith a vitality sensor to measure pulse re-sponse. In addition there is growing interestin brain computer interfaces (BCI) as illus-

trated by the instigation of an annual Neu-roGaming Conference and the developmentof technologies such as Foc-Us,a headsetfor gamers, which delivers transcranial Di-

rect Current Stimulation (tDCS) with theaim of increasing brain plasticity to improvegame-play performance, and MUSE,a

brain-sensing (EEG)headband designed toallow some degree of brain control, whichhas been developed with a wide range of ap-plications in mindincluding gaming. Te introduction of biometric interfacesand sensors of this type represents a signifi-cant departure from body motion detectionsystems, heralding a future in which complexphysiological data, such as skin response,

heart rate, respiration, brain response, andso on, might be captured and used to manip-ulate and customize interactive game-playfor the user. Such biosensors would create afeedback loop, constantly measuring physio-logical data induced by game-play and usingthis data to trigger further physiological andemotional responses. If the games industry is tottering on the

brink of a new chapter in its development,one in which the voluntary and involuntaryphysiological responses of players becomethe primary focus of interface development,

Plug-In to Fear: Game Biosensors

and Negative Physiological Responses to Music

Abstract.Te games industry is beginning to embark on an ambitious journeyinto the world of biometric gaming in search of more exciting and immersivegaming experiences. Whether or not biometric game technologies hold the keyto unlock the ultimate gaming experience hinges not only on technological ad-vancements alone but also on the game industrys understanding of physiologicalresponses to stimuli of different kinds and its ability to interpret physiological datain terms of indicative meaning. With reference to horror-genre games and music

in particular, this article reviews some of the scientific literature relating to specificphysiological responses induced by fearful or unpleasant musical stimuli andconsiders some of the challenges facing the games industry in its quest for the ulti-mate plugged-in experience.


3/22

. /

it stands to reason that the industry willalso require a good understanding of phys-iological responses induced by game-play.Tis would prove to be quite a challenge.

Te gaming experience is made up of manyinteracting elements, each potentially capa-ble of inducing some sort of physiologicalresponse, either as individual elements orworking in conjunction with one another.However there is one element that gamedevelopers would recognize as of particularsignificance for its potential to manipulateplayers responses, and that is audio (music

and sound). Recent years have seen a significantnumber of scientific studies exploring theresponses of test subjects to different types ofmusic and sound. Te scope of these studiesextends beyond the parameters outlined sofar, but clearly studies of this kind providea valuable and continually growing referen-tial resource for anyone seeking to explore

the connection between music, sound, andmeasurable physiological data. However, inorder to tailor such data to the needs of thegames industry, it would be necessary tofocus on responses to those musical char-acteristics most commonly associated withparticular game genres. Tis at least wouldprovide game developers with some tentativepredictive outcomes when designing proto-

type biosensor interface systems. Of all the games available to date, hor-ror-genre games are perhaps the most likelyto induce the strongest and most visceralphysiological responses. In a series of scien-tific tests designed to measure physiologicalresponses to various fear inducing games,Dead Space was found to trigger, multipletypes of fear response.Te game proved to

be universally frightening regardless of theparticipants previous gaming experience.Clearly, allthe elements of the game work insynergy to induce responses of this kind, butit is undeniable that the games use of avant-garde, aleatory music plays a significant role,

triggering and sustaining fear responses.Jason Graves, the composer of the musicfor the Dead Spacefranchise, describes themusic for the games as non-musical.Tere

are a number of reasons why this descriptionappears to fit: minimal thematic materialfeatures in either of the Dead Spacegames;the aleatory approach results in a lack ofrecognizable patterns or predictable events,creating a persistently dissonant and exper-imental sound world of constantly shiingtextural layers and unexpected, randomizedstingers,all responding in real-time to

interactive game-play. In order to discover possible musical trig-gers for the types of fear responses recordedin the Dead Spacestudy, it is necessary toreview the scientific evidence linking musicto negative physiological responses. Tisinformation might then be applied withinthe context of wider games development;anticipating likely physiological responses,

predicting and measuring biosensor data,and then applying this within real-timegame-play. Tis article presents some of the scientificfindings linking specific negative physiologi-cal responses to musical stimuli and seeks tocontextualize these results in a preliminaryoverview of some of the complications pre-sented by the scientific literature currently

available. Tere follows an exploration of thepotential challenges faced by those in thegames industry tasked with utilizing suchphysiological responses in the creation ofentertaining, biosensor-enhanced gamingexperiences. Te discussion concludes bysuggesting possible benefits of music-relatedbiosensor developments in wider fields.

Reviewing the Scientific DataACautionary Introduction

When reviewing the expansive body of sci-entific literature currently available, it quicklybecomes apparent that there is much we still


4/22

: Game Biosensors

do not understand regarding the complex in-terplay between physiological and emotionalresponses such as fear, particularly within thewider context of everyday life. Results can

be highly inconsistent and contradictory, soat best the literature shows possible trends,generally within controlled conditions. Inaddition, where connections between physi-ological and emotional responses are found,it can be difficult to determine which comesfirst, since physiological responses can bethe product of, and trigger for, subjectiveemotional feelingsa process known as pe-

ripheral feedback.

For these reasons, it canbe difficult to infer emotional meaning fromphysiological responses. Te literature also raises questions aboutthe nature of emotions, whether perceived orfelt, and the language or terminology used todescribe them. Emotive words such asfear,anger, anxiety, and so on, lack scientific pre-cision and clear differentiation and cannot

be universally applied across studies involv-ing animals as well as humansat least notwithout resorting to anthropomorphism. Forthis reason, physiological and emotional re-sponses are oen described in terms of de-fensive and appetitive motivational systems.Responses to negative emotions in particularare frequently linked to survival and defencemechanismsthose activated in the pres-

ence of a threat. Tis potentially raises problems when ex-ploring negative physiological responses tomusical stimuli, since music presents no sur-vival threat, however fearful our perceptionof it might be. Questions inevitably followregarding the nature of music-induced emo-tional responses.It might be logical toquestion whether or not emotions reported

by participants when listening to fearfulmusic represent genuine feelings and expe-riences of fear, or only theperceptionof themusic itself as fearful or scary. Nevertheless the scientific literature doesshow a connection between measurable

physiological responses associated with emo-tional processing, or appetitive and defensivesystems, and musical stimuli. However com-parative reviews of the resulting data present

a number of challenges. For example, in thecase of studies exploring musically inducedfear responses, one of the primary triggersfor negative physiological responses, of thekind normally associated with unpleasant-ness, aversion, fear, and so on, is dissonance.However, determinations of dissonance,as exemplified in scientific studies, differsignificantly. ypes of musical stimuli used

to test responses to dissonance vary consid-erably, from consonant excerpts with a fewdissonant elements,or processed excerptsin which the melody lines have been trans-posed against unaltered accompaniments,through to music specifically composed tobe atonal and dissonant throughout.Inaddition, stimuli durations also vary signifi-cantly from study to study. Tese differences

of approach have a direct bearing on theresults reported in these studies and make itdifficult to evaluate the data in general terms. For example, in the case of pitch-manipu-lated extracts, the unalteredmusical featureshallmark the stimuli with the stylistic finger-prints of tonal music, creating expectationswhich are constantly frustrated by artificiallycreated dissonances. In such cases, it becomes

difficult to determine whether or not thephysiological responses reported are triggeredby dissonance alone, when in fact they mightbe the product of perceived musical error, orfrustrated expectations. Musical examplessuch as these identify themselves as abnormalor unnatural and therefore are potentiallyproblematic when used as isolated examplesof dissonance. Interestingly those studies that

utilize musical stimuli of this type appear notto recognize this issue, as can be seen in state-ments such as the following:

Importantly, consonant and dissonantversions of each piece were, thus, identical


5/22

. /

in their dynamic outline, their rhythmicstructure and tempo, and their melodiccontour. Consequently, it is not possiblethat any of these stimulus properties con-tribute to differences in autonomic andneural effects between consonant and dis-sonant pieces.

Another potential problem encounteredwhen reviewing biofeedback literature is thatalthough the musical features vary consider-ably, in some cases researchers fail to providesufficiently detailed information regardingthe musical stimuli used, simply describing

the music in terms of classifications such asarousal and valence,or providing the titleof a work and its composer without detailingthe specific extracts used.Tis lack of mu-sical clarity and precision can undermine thepotential to differentiate between the musicalfeatures or elements responsible for inducingspecific physiological responses. For this reason, the following biofeedback

literature review is structured according toresponse typerather than being organizedaccording to specific musical features. Whereappropriate, endnotes identify details ofthe musical stimuli used in order to clarifyterms, such as dissonance, used in the stud-ies reviewed.

Brain Response

Within the context of emotional processingand responses to musical stimuli, there aremany scientific studies that explore the brainand its interactions with other body mech-anisms. Tese studies identify a numberof structures and regions within the brainthat are important in terms of appetitiveand defensive responses. Te amygdalainparticular emerges as critical to emotionalprocessing, experiences of fear, as well as tofear and defense conditioning in responseto external stimuli. In addition, responses tofearful stimuli implicate the amygdala withother bodily responses (endocrine, auto-

nomic, and behavioral),such that threats. . . not only elicit specific defense responsesbut also initiate generalized arousal.

A number of music-based studies have

served to extend such investigations, utilizingdissonant musical stimuli as a means of in-vestigating the response of the amygdala andother brain structures to unpleasant, aversive,or fearful musicfounded on the basis of agrowing and substantive body of evidencesuggesting that dissonance elicits unpleasantjudgements in normal listeners.For exam-ple, one FMRI studyfound that dissonant

music stimulated increases in blood oxygenlevels in the amygdala, the hippocampus,the parahippocampal gyrusand the tem-poral poles:interestingly the very regionsthat showed strong decreasesin blood oxygenlevels in response to pleasant music (theseobservations are paralleled in other studies).

Evidence such as this has been corrob-orated by research testing the response of

patients following surgery or damage to thebrain. For example, it has been found thatbilateral damage to the amygdala impairsthe perception of fear, facial expressions, andunpleasant emotions. Significantly it alsoimpairs the recognition of scary music.

Similarly, damage to the parahippocam-pal cortex(or PHC) has also been foundto affect the perception and evaluation of

dissonance. One study found that patientswith larger excisions of the PHC appear toshow increased indifference to dissonance.Te authors of the study also implicatethe parahippocampal gyrus (or PHG) inthe avoidance of aversive stimuli. So whatemerges from the study is a picture in whichthe amygdala and the PHC evaluatestimulias aversive, while the PHG is then engaged to

protect by seeking to avoidaversive stimuli. Interesting though such results might be,it is unlikely that commercial gaming bio-sensors, with the capacity to measure directresponses in the amygdala or hippocampus,will appear any time soon. Even if such gam-


6/22

: Game Biosensors

ing biosensors did exist, biometric data gen-erated in response to dissonant music mightnot indicate a fear response, as Koelschpoints out:

Activity changes observed in the amygdalaand the hippocampus during the presenta-tion of unpleasant (or threatening) stimuliis not necessarily a result of the generationof fear (or other unpleasant emotions)but could well reflect inhibitory processesactivated to prevent the hippocampusfrom traumatization during exposure topotentially harmful stimuli.

In other words, some of these responsesmight simply be the result of self-protectionmechanisms at work within the brain. Nev-ertheless, the activation of defensive mecha-nisms in response to threat detection mightwell act as a prompt to associated emotionalprocessing. If the responses of brain structures suchas the amygdala and hippocampus cannotcurrently be accessed directly in commercialgaming contexts, how might brain responsebe captured? Given recent technological de-velopments,it seems likely that the gamesindustry will initially focus its energies onmonitoring electrical activity in the brain(EEG measurements). Commercially avail-able EEG devices already use such data(detecting alpha and beta waves) to indicatedifferent states such as relaxation and con-centration. However, to capitalize on EEG develop-ments in gaming, it is likely that the industrywould wish to extend the interpretive scopeof such physiological data beyond simply de-tecting levels of concentration or relaxation.A significant number of studies have usedEEG data as a measure of emotional andphysiological response, and these could serveas a springboard to more specific game-related research in the future. In one particularly interesting music-based study, researchers explored EEG and

fMRI data in conjunction with subjectiveemotional evaluations in response to musictaken from the atonal and athematic scorefor the film Danton(), composed by Jean

Prodromids. In contrast to musical stimuliused in many other studies as a stimulantto fear, this music was specificallycomposedto underscore dark and emotive onscreenevents;and it belongs to a sound worldmore closely aligned with horror-genregames such as Dead Space. It is interestingtherefore that the authors found that acti-vation areas involving right cortical regions

were found only for unpleasant emotions,and go on to suggest that the predominanceof le hemisphere activation with pleasantmusical feelings and right activation withunpleasant ones is consistent with findingsthat relate right frontal activation with neg-ative affect and le frontal activation withpositive affect.

If the right and le hemispheres activate

differently in response to unpleasant andpleasant musical stimuli, and these responsescan be measured via EEG, then the implica-tions of such research could be significant interms of gaming applications. However, thestudy does raise questions regarding the stim-uli used. For example, it is not clear whichextracts of the original film score were used,which makes any analysis ofspecificmusical

features and associated appraisals of unpleas-antness or fear impossible. Unresolved disso-nance is a general characteristic of the entirescore, and based on other studies, this is amusical feature strongly linked to unpleasant-ness and negative emotional and physiologicalresponses. Te authors of the study also pointout that Prodromids breaks classical musi-cal grammar rules, frustrating expectations

in terms of melody and harmony, so trig-gering the participation of limbic and righthemisphere regions in the search for emo-tional comprehension.

However, other factors might play a partin the responses found: some sections of the


7/22

. /

score make extensive use of disturbing vocaltextures and pitch oscillations, creating anaural evocation of tortured humanity. If theextracts were taken from passages such as

these, might the participants responses bedue to their reactions to alarming and ap-parently distressed voices, rather than othercompositional and instrumental features? Nevertheless, the findings of the study areof relevance because they suggest that it ispossible to differentiate patterns of electricalactivity in specific brain regions in responseto musical stimuli designed to trigger polar-

ized emotive responses. Clearly, however,challenges remain regarding the identifica-tion of precise characteristics of the musicalstimuli used and the extent to which thesecan be linked to specific emotional and phys-iological responses. In addition, the resultsdo not account for the additional cognitiveprocesses involved in game-play, or for thebombardment of the senses by simultaneous

stimuli while gaming. However, they do in-dicate avenues to explore in the developmentof EEG-based gaming devices. EEG devices may represent the mostlikely avenue for the detection of directbrain response in game-play scenarios, butwhat of other aspects of brain response,such as coordinating the activity of vari-ous body systems? Some studies frame the

brains emotional response within the con-text of survival circuits.In laymans terms,these survival circuits might be describedin terms of thriving and survivingwithcircuits involved in defense, energy main-tenance, fluid balance, nutrition supplies,thermoregulation, and reproduction, toname a few.Tese circuits coordinate thebodys integrated systems and behavioral

interactions with the environment. For example, if the specialized defense cir-cuits of the amygdala detect a threat, a num-ber of consequences follow: hard-wiredbehavioral reactions occur; relevant goal-

directed actions are learned and performed(or stimulus-response habits activated); andarousal responses are triggered in the centralnervous system(CNS), the autonomic ner-

vous system

(ANS), and endocrine system.Te result is a heightened state character-ized by increased attention and sensitivityto stimuli, linked to associated functions oflearning, memory formation, and retrieval.Tis in turn creates a feedback loop, whichcontinues to drive the survival circuits. Given the connection between survivalcircuits in the brain and generalized arousal

in the body, it would seem logical to de-duct that the games industry might exploitphysiological responses, such as those ofthe autonomic nervous system, as indirectindicators and triggers of brain response.Generalized arousal responses have alsobeen linked to a number of theories ofemotional processing,potentially present-ing the games industry with indicators of

emotion as well. A number of these theoriespresent the autonomic nervous system as amajor component of emotional response,appearing to confirm the potential of ANS-response biosensors as possible indicatorsof emotion. Te road to a full scientific understandingof the complex neural systems of the humanbrain and its response to music and emotion

still stretches out a long way before us. Un-doubtedly there is evidence to support theassociation found between dissonant mu-sical stimuli and unpleasant, negative, andpotentially aversive responses.Accessingassociated brain responses directly, however,presents very real challenges to understand-ing and to practical and technical imple-mentation. In terms of directaccess to brain

activity, EEG-based devices offer the mostfeasible future pathway, but the responses ofthe autonomic nervous system, such as heartrate and skin conductance, potentially offerless difficult terrain to explore.


8/22

: Game Biosensors

Heart Rate and Skin Conductance

A number of studies confirm that listening tomusic has an effect on the ANS.If the ANS

is a component of emotional processing, ashas already been suggested, then it is logicalto deduct that these physiological responsesmight be indicative of music-induced emo-tions. Whether or not it would be possible todifferentiate specific emotions such as fear,based on distinct autonomic signatures, is amore controversial matter.

Sylvia Kreibigs detailed review of liter-

ature exploring ANS activity and emotionnotes that many studies report increases inheart rate and skin conductance in responseto fearful stimuli, indicating heightenedarousal.Other studies report similarresults.However, when reviewing mu-sic-based studies, notable exceptions soonbecome very apparent. A significant num-ber of these show that heart rate decelerates

significantly in response to unpleasant,dissonant musical stimuli. In addition, skinconductance tends to heighten as levels ofdissonance increase.(Interestingly thesame results have been reported in responseto negative, emotional films, pictures, andsounds.) Some have tried to explain theseresults,but they do appear to be counterin-tuitive.

Heart rate accelerations (tachycardia) anddecelerations (bradycardia) appear to betriggered by a number of musical character-istics such as rhythm, texture, and dynamics.Some studies have sought to analyze theserelationships more closely, attempting toidentify specific psychoacoustic featureslinked to particular levels of arousal, valence,and emotional states.

One particularly interesting study byEduardo Coutinho and Angelo Cangelosiselected six psychoacoustic features (loudness,pitch level, pitch contour, tempo, texture, andsharpness), and used these to create a predic-

tive computer model of emotional responsebased on arousal and valence.Tey foundheart-rate changes linked to specific soundfeatures, such as increasing loudness

indicating higher arousal. Heightened arousalwas also associated with higher pitch, fastertempi, and sharper sounds. Valence correlatedwith tempo and pitch levels.

It is difficult rationalizing some of theseresults with the findings of other studiesshowing heart rate deceleration in responseto fearful musical stimuli, since some of thefeatures identified with heightened arousal

are also associated with horror-genre music.Clearly other factors are also at work. For example, a study by Bradley and Langsuggests that the presence of pleasant andunpleasant sounds has an impact on heart-rate fluctuations. Tey found that

heart rate deceleration was significantlygreater when listening to highly arousingunpleasant sounds . . . than when listen-

ing to unpleasant sounds that were ratedlower in arousal. . . . For unpleasant arous-ing sounds, the largest heart rate decelera-tions were obtained for a male scream . . .a female scream . . . and sounds of attack.

Given that nonmusical sounds oen fea-ture in horror-genre music, there is certainlyscope to activate heart-rate decelerations as-

sociated with highly arousing and unpleasantsounds. It is also likely that the wider soniccontext of horror games would also be acontributing factor in this regard, providingdisturbing and fearful sound effects with thepotential to intensify any negative physiolog-ical impacts triggered by the music. Interestingly, however, there is also evi-dence suggesting that unpleasant aural stim-

uli might prime people to be startled andintensify the effect when it occurs. One studythat measured the magnitude of startle re-sponse by measuring blink reflex found thatlarger blinks elicited to visual startle probes


9/22

. /

when listening to unpleasant compared withpleasant sounds . . . [and] . . . blink reflexeswere faster when listening to emotionallyevocative sounds.Tese reactions were

accompanied by greater activity of the facialmuscle associated with frowning (corruga-tors supercilli). Another intriguing findingof this report was that high arousal soundstriggered a strong memory response.So itwould appear that unpleasant aural stimulican act as a primer for fear (intensifying andspeeding up reactions), ensuring that onceexperienced, the fear is not forgotten.

Significantly, however, the startle reflexis characterized by an increasein heart rate.Music written for stingers (sudden onsetcues or hits designed to surprise) plays tothe startle reflex, so given the frequent useof stingers in horror-genre music, one mightexpect to see significant heart-rate accelera-tion. Within the context of horror music, thisis not the case. One intriguing study found

that suspenseful [horror] music works as amild pre-pulse and therefore limits the effec-tiveness of a startle that follows it,effec-tively making the stinger moments less scary.Ironically it is the fearful horror context thatinhibits cardiac acceleration. Another factor influencing heart-ratefluctuations appears to be the subjects famil-iarity with, or knowledge of, music, as re-

ported in a study by J. E. and H. F. Landreth:Heart rate response appears to be linkedwith the presence or absence of learning andrepetitive exposure to music. . . . Results ofthis study demonstrate that physiologicalresponses of the listener can be affected byhis increased knowledge and understandingof a musical score.Findings such as thismight be of particular significance in relation

to repeated game-play; indicating that overtime, physiological responses to games po-tentially change as players become increas-ingly familiarized with musical stimuli andlearn what to expect. In addition, if playersare already familiar with the musical sound

world of particular games due to widerlistening or educational experiences, thentheir physiological responses might be verydifferent from those of players who are less

familiar with those musical genres. Famili-arity is linked to expectation to some degree;if we are familiar with something and haveknowledge of it, we know what to expect,which in turn affects physiological response.Conversely, the same applies with regards tothe unfamiliar and unexpected. It is thought that significant and novelstimuli trigger orienting responses, engaging

motivational circuits and prompting the dis-position to act. Some physiological responsesare considered to be indices of orientingand attention, governed by appetitive anddefensive neural motivational systems.Significant stimuli might be defined in termsof pleasure and arousal, suggesting linkswith motivational systems associated withemotional processing and also task-relevant

contexts. Novel stimuli tend to be linkedto danger and the engagement of defensesystems.Te degree of threat, arousal, andnovelty presented by the stimulus modulatesresulting responses. It might be logical to assume, therefore,that novel stimuli might trigger cardiacacceleration; however, the reverse is true.Significant cardiac deceleration occurs in

response to novel stimuli, particularly inthe case of aversive and unpleasant cuesindicating enhanced sensory intake andorienting. Skin conductance changes are alsoheightened in response to novel stimuli. Studies suggesting that sudden musicalchanges induce cardiac deceleration appearto be consistent with orienting responsesto novel stimuli. For example, one study

found that musical segments with eminentchanges in rhythm, texture, and dynamicsproduced bradycardia in the experimentalsubjects,perhaps indicating that suddenjuxtaposed contrasts trigger orienting re-sponses associated with novel stimuli. Te


10/22

: Game Biosensors

music of Dead Spaceand other horror-genregames and films is oen characterized bydramatic and unpredictable musical changesand unexpected sonic events. On this basis,

orienting responses to novel stimuli mightbe tentatively proposed as one factor in theconnection between fearful musical stimuliand cardiac deceleration. However, orienting responses to novelstimuli are fairly short-lived. Physiologicalchanges are most noticeable in the first fewseconds following the stimuli presentation.In addition, repeated exposure to the stimuli

presented appears to lessen the orientingresponse (heart rate habituates fairly rapidly;skin conductance habituates more slowly).Tis raises questions regarding the ways inwhich physiological responses change overtime, particularly with regard to fearful mu-sical stimuli, and confirms the significance offamiliarity as an influencing factor in physi-ological response. Whether or not prolonged

or repeated exposure to horror-genre musicwould result in similar habituation, particu-larly within the context of dynamic mixing,would be a matter of some significance inbiometric gaming applications. Te connection between orienting re-sponses and novel stimuli only explainsrelatively momentary physiological changessuch as cardiac deceleration and heightened

skin conductance. If there is a link betweenorienting responses and eminent musicalchanges, such a link would not explain thefindings of studies showing more prolongedexamples of lowered heart rate or increasedskin conductance in response to fearfulmusic. Tere are, however, a number of studiesthat appear to indicate connections between

fearful anticipation and suspense, or increas-ing unpleasantness over time, and cardiacdeceleration and increased skin conduct-ance. One particularly interesting film-basedstudy demonstrated that skin conductanceincreased during longer periods of antici-

pation leading up to an unpleasant accidentscene.Te results indicated that suspensewas more disturbing than surprising.Tisfinding is consistent with other research

indicating that physiological responses in-crease over time.For example, one studyreported a significant correlation showingthat . . . heart rate deceleration increasedlinearly with increasing unpleasantness.In a review article, Koelsch reports thatthe strongest physiological effects for eachemotion type generally tended to increaseover time, suggesting that the intensity of

an emotional experience is likely to increaseover time during the perception of a musicalexcerpt.

If, as the evidence indicates, long periodsof anticipation induce significant physiolog-ical stress reactions and emotional intensityincreases over time, these findings wouldappear to contradict the opinion expressedby Teodor Adorno and Hanns Eisler in

relation to suspenseful anticipatory cues:If the screen shows a peaceful countryhouse while the music produces familiarsinister sounds, the spectator knows at oncethat something terrible is about to happen,and thus the musical accompaniment bothintensifies the suspense and nullifies it bybetraying the sequel.In one sense this istrue, the music does let the cat (or, in the

case of horror, monster) out of the bag,but the physiological evidence suggests thatknowing the monster is going to be let out ofthe bag and anticipating or imagining whatmight happen can be far more stressful thanbeing faced with the surprise of it jumpingout suddenly. In one sense, however, Adornoand Eisler were right: since if the sequelwere to be accompanied by a stinger mon-

ster cue, the suspenseful horror-genre musicwould inhibit the physiological impact of theresulting startle reflex.

Discussions regarding the respective phys-iological effects of suspense versus surprisepivot around the concept of expectation.


11/22

. /

Scientific literature points to a number ofmusical characteristics governing expec-tancy, such as melodic interval size . . . me-lodic contour . . . rhythmic features . . . and

tonal and harmonic structures.

Harmonicviolations appear to have attracted the great-est share of research attention. For example,in drawing together the findings of a numberof research studies, Stefan Koelsch assertsthat

unexpected chords . . . violate the sensoryexpectancies of listeners . . . [and] theviolation of musical expectancies has been

regarded as an important aspect of gener-ating emotions when listening to music.. . . Moreover, the perception of irregularchord functions has been shown to leadto an increase of perceived tension . . . andthe perception of tension has been linkedto emotional experience during musiclistening.

Tese findings are shared by another report,

also published in , that states, It can beargued that . . . [harmonic] violations lead toan increase in physiological arousal, whichin turn leads to a heightened emotionalexperience of the musical pieces. Furtherthe authors suggest that there is compellingsupport for the idea that suspensions of mu-sical expectations are an important pathwayto generating emotions in the listener.

One study that attempted to explain theconnection between frustrated harmonicexpectation and physiological responses sug-gested that the number of perceived pitchescommon to successive chords might be onefactor governing musical tension: the weakerthe pitch commonality values, the greater theperceived tension. Further, the authors sur-mised that horizontal motion might also

be significant, indicating that the larger themelodic interval in each voice, the greaterthe perceived tension.

Te power of unexpected musical or sonicevents of any kind (whether harmonic or

not) to trigger physiological responses isconsistent with studies linking musical char-acteristics to fear vocalizations. For example,with reference to a study by Arne hman,

Klaus Scherer suggests that musical stimulisharing the acoustic characteristics of . . . fearvocalisations (sudden onset, high pitch, widerange, strong energy in the high frequencyrange) may be appraised by . . . extremelypowerful detection systems and may provoke. . . physiological defence responses.

Whatever the basis for our physiologicalresponses to unexpected musical events, to

suspense and dissonance, or to any othermusical or sonic characteristics for thatmatter, one fact emerges unequivocally fromthe evidence, namely the physicality of ourresponse to aural stimuli, whether in thebrain or the body. More specifically, researchindicates that musical or sonic characteristicscommonly associated with musical portray-als of fear, horror, evil, and so on (in the pop-

ular media), such as unresolved dissonanceand those already outlined, tend to induceresponses more normally associated withunpleasant or stressful stimuli. Te evidencealso stresses the importance of the widerextramusical context in determining phys-iological responses: for example, the soniccontext (the affect of unpleasant, nonmu-sical sounds); the visual context (the affect

of visual stimuli such as film); the personalcontext (the familiarity of the subject to thestimuli). Whether or not such associationsare biologically predetermined or acquired,as a product of social and cultural condition-ing, is difficult to ascertain.

Nature or Nurture?

Scientific data throws some light on this

question, but it is not unequivocal. A num-ber of studies have shown that young infantsshow a marked preference for consonant asopposed to dissonant music. A studyreported that babies as young as two months


12/22

: Game Biosensors

old exhibit a preference for consonance.Te results confirmed the outcomes of anearlier study by rainor and Heinmillerexploring the development of evaluative

responses to music, which had found thatinfants aged four to six months preferredto listen to consonance as opposed to dis-sonance.Research by Zentner and Kagan,examining infants perception of consonanceand dissonance in music, further supportedthese findings.Te rainor and Heinmillerstudy reports that although infants do notyet have the musical-system-specific knowl-

edge of scale structure that is involved inadults emotional reactions to music, infantsare similar to adults in their evaluative reac-tions to consonance and dissonance.Teauthors state that infants can also discrim-inate consonant from dissonant intervals,and cite a number of articles to support thisclaim.Further they suggest that

because -month-old infants do not yethave knowledge of musical scale structure. . . emotional reactions to music cannotarise in an identical manner in infantsand adults. . . . Tus it is particularly in-teresting for the study of the origins ofemotional responses to music that infantsshow similar affective reactions to thoseof adults to consonance and dissonance.It indicates first of all that infants have

affective responses to an aspect of musi-cal pitch structure early in life. It alsoindicates that the consonance/ dissonancedimension is very basic to musical pro-cessing.

If this is indeed the case, can we feelconfident asserting that responses to dis-sonance are innate and not learned? Notaccording to the authors of the study,

who write, It remains an open questionwhether -month-old infants preferencefor consonance is a direct consequence ofauditory system structure or whether it islearned early in life through exposure to

sounds in the natural environment.If apreference for consonance is not innate butlearned, the onset of the learning processis occurring remarkably early and is taking

root very quickly. Innate or learned, thereis no denying that consonance is highlysalient to infants.Might the speed of thelearning process be linked to an innate orlearned association between dissonance andfear or danger? Unfortunately the learnedversus innate debate also applies to asso-ciations linking dissonance to fear or dan-ger. If learned, it has been suggested that

the learning process occurs from an earlyage, rooted in the infants response to themothers voice expressing disapproval.Ina review of studies relating to vocal expres-sion and musical performance, Juslin andLaukka similarly suggest that music per-ceived as scary may mimic characteristicsassociated with vocal responses to danger.However this association is acquired in

children, it is a fact that the perception ofdanger links music with biologically impor-tant functions.

Whether preprogrammed into the fabricof brains and bodies or acquired rapidly dur-ing early development, associations betweenperceived danger and physiological responsecan be difficult to disconnect in adulthood,influencing not only our perception of dan-

ger but also our perception of other people.For example, a recent psychology studyfound that thriller music tended to inhibitaudience identification with onscreen char-acters, making them appear less likeable andtheir thoughts more uncertain. Te musicdiscouraged empathy and identification,encouraging viewers to distance themselvesfrom the potential danger of unreadable and

unpredictable people; effectively ostracisingthem as other.

Viewed collectively the scientific evidenceshows that from the earliest age, music canstimulate a range of different physiological


13/22

. /

responses, which in turn have the powerto influence perception and associated be-havior. More specifically, the evidence indi-cates strong connections between musical

characteristics commonly associated withdepictions of fear and horror, such as thosefound in the Dead Spacefranchise, and nega-tive physiological responses more normallyassociated with fear, stress, unpleasantness,and aversion. Te commercial success of theDead Spacegames indicates that the abilityto induce real fear, as evidenced by corre-sponding physiological responses exhibited

during game-play, might carry a significantfinancial incentive. However, might we notassume that the generation of any strongphysiological responses, whether positiveor negative, could potentially indicate likelycommercial success and healthy sales? Tisstudy has focused on negative physiologicalresponses, but for every piece of scientificdata revealing the links between music and

negative physiological responses, there isother data showing the potential of musicto trigger positive physiological responses.Games designed to exploit these responsesmight well create such positive, feel goodexperiences that they translate into furthersales. Against this backdrop, the develop-ment of biosensor interfaces appears to makecommercial sense for the games industry,

while also offering the potential to cre-ate more exciting and responsive gamingexperiences as well. However, the scientificresearch also points to a number of problemsfor game developers seeking to capture andexploit physiological data of this type.

The Challenges

Perhaps one of the most significant chal-

lenges facing game developers is the acquisi-tion of a comprehensive understanding ofthe ways in which the body and brain re-spond to different types of music and sound.Significant strides have been made by the

scientific community in recent years withregards to understanding our responses tomusic and sound, but this field of scientificendeavor still presents many unanswered

questions. Difficulties of understanding arefurther compounded by the fact that game-play itself is not simply a passive experiencein which one listens to music in isolation;the music, as in film, performs dramaticfunctions relating to visual stimuli and nar-rative contexts, which in turn also triggerphysiological responses. Unlike film, theseexperiences then influence the players in-

teraction with the game, further affectingtheir responses. Scientific data relating tomusic alone, or even music and film, al-though potentially informative, would notfully address the complexity of the activegaming experience. Te development ofgames utilizing biosensor data to create aninteractive feedback loop, measuring physi-ological responses and then adapting game-

play to affect those responses even more,would further compound the challenges forthose seeking to understand the relationshipbetween music, sound, and physiologicalresponses within the context of game-play.It could be argued that without an adequateunderstanding of these complex relation-ships, and with many questions still waitingto be addressed by the scientific community,

the potential of biosensors to enhance thegaming experience could be compromised,and in worst-case scenarios, poorly imple-mented biosensor data might even detractfrom the gaming experience. Even if the games industry honed theirapproach and focused on developing biosen-sor systems utilizing biofeedback data fromone type of physiological response only, such

as heart rate, problems would still remain.For example, the challenge of isolating reli-ableindividual physiological responses onthe basis of biofeedback literature examiningmultiple responses and interactions betweenvariables of response, general patterns, and


14/22

: Game Biosensors

trends would be significant. It is unlikelythat individual biosignals could be read asreliable or universal emotional indicators,particularly within the real-time context of

interactive game-play. In the case of skin-conductance response,for example, individuals can have very differ-ent levels of response. Some people (labelledas labiles) exhibit significant responses evenwhen at rest, others (stabiles) have skinresponses with very little variation whenat rest. It is thought that such variations inbaseline response might be linked to person-

ality type. Whatever the underlying causesmight be, the evidence points to significantvariations in individual response; in otherwords, it is not simply a case of one shoe fitsall. Should questions of reliability of responsebe successfully addressed, other challengeswould still remain. For example, a simplebiosensor designed to measure heart rate

alone might detect a drop in heart rate dur-ing a particularly frightening sequence ofgame-play in response to disturbing music,sound, and graphics. Unless the game devel-oper is aware that this response potentiallyindicates a negative, stress response, the datamight be interpreted within the program-ming of the game as an indicator of fallingtension. Te misinterpretation of this data

could potentially result in inappropriate ad-aptations within the game. Understanding,therefore, has a direct bearing on successfuldata integration and implementation, as wellas biosensor design and development. If individual physiological responses canbe unreliable and misunderstood, it mightbe logical to assume that the developmentof interfaces with multiple sensors, perhaps

measuring skin conductance, and even brainresponse, as well as heart rate, might providegreater clarity in terms of overall response.Utilizing comprehensive arrays of physiolog-ical measures to detect patterns of responseand associated variables is an approach

advocated by many scientific studies.Forexample, in the gaming scenario describedabove, heightened skin conductance mightprovide a more reliable and less unexpected

measure of physiological response, whichmight help to contextualize heart-rate data.However, the development of multiplebiosensor systems would not be withoutchallenges: data from different sensorsmight appear to be contradictory, creatingcomplications of interpretation and integra-tion; processing complex data streams andsuccessfully implementing them within the

interactive environment of real-timegame-play would be technically and creativelychallenging; finally, designingpracticalmul-tisensor systems for home use, which wouldbe capable of measuring the complexities ofbrain response, would be difficult. For composers of music for games, sys-tems such as these, requiring some under-standing of the musical triggers responsible

for specific physiological responses, accom-panied by a grasp of the ways in which theseresponses might be musically manipulated,represent a significant challenge. As it stands,composers already face challenges in termsof creating music suitable for use in theinteractive environment of game-play anddynamic real-time mixes. In addition, theirlevels of involvement in the integration pro-

cess vary significantly, potentially creatingfurther complications. In a new dawn of biosensor interfaces,few composers would be aware of findingsin scientific literature linking specific musi-cal or sound-based stimuli to physiologicalresponses; even if they had the inclinationto explore such literature, it is unlikely thatthey would have the time to do so, particu-

larly given current commercial productionschedules. Most composers would base theirmusical depictions of particular emotionsor dramatic situations on simple gut feel-ings about what would be most musicallyappropriate. It might well be that these gut


15/22

. /

feelings result in music that stimulates thehoped-for physiological responses, but thenagain, it might not. In an ideal scenario, it might be desirable to

be able to predict the physiological responsesfor individual musical cues, whether expe-rienced individually or layered together in adynamic mix with other randomized musicalor sound-based elements. Once the result-ing data is received and processed, musicaladaptations might occur to manipulate theseresponses in knownways. Whether or not thisscenario is achievable hinges on a number of

factors: the levels of understanding regardingthe relationship between music and physi-ological responses, or the accuracy of the gutfeeling approach as a trigger for predictableresponses; the potential of music to be trulyadaptive in response to complex and continu-ally changing real-time data streams; andfinally, the ability to address the technical andcreative challenges relating to integration.

It might well be that efforts to answerchallenges such as these result in a movetoward algorithmic composition in whichprecomposed music is discarded in favor ofmusic generated and adapted in real-time inresponse to continual data feedback, basedon known relationships between specificmusical or sound-based stimuli and physi-ological responses. Given that precomposed

music cannot be truly adaptive, this outcomemight appear to offer a solution. However,substituting the role of the composer for aset of algorithms (based on scientific data)is unlikely to produce the most satisfactorymusical results, regardless of the physiologi-cal responses induced. Alternatively, composers might work ac-cording to guidelines set in place by game

developers; guidelines based on scientificevidence linking specific musical triggers toassociated physiological responses. A differ-ent approach might be to develop predic-tive computer models of physiological andemotional response, based on scientific data

showing generalized patterns and trendsthose most likely to be triggered by specificmusic elements or sound-based characteris-tics. Such models might then be integrated

alongside existing industry middlewaretools,providing composers and sounddesigners with predictive data, which inturn might guide their creative work towardthe achievement of specific and anticipatedphysiological responses. It might also be pos-sible for the industry to release gaming prod-ucts with adaptable and intelligent predictivegaming models, which incorporate some

degree of customizationsuch that general-ized data would eventually give way to theunique response profile of individual gamers. Te development of predictive computermodels such as those created in the studiesby Coutinho, Cangelosi, and Dibben pos-sibly suggest a way forward for the gamesindustry, particularly given the surprisinglevels of accuracy shown.If as the authors

conclude, a significant part of the listenersreported emotions can be predicted from aset of six psychoacoustic features,then itis not difficult to see the potential for predic-tive models in biometric gaming. Putting aside the obvious difficultiesinvolved in creating fully comprehensivepredictive models or guidelines, it is unlikelythat composers would welcome anything

that might constrain their creative approach.Even if computer models and guidelinessuch as this were instigated and standardizedacross the industry, it is likely that strikinglysimilar musical characteristics would beshared across musical depictions of similardramatic or emotional contexts. Te dangerof standardization in this context would bepredictability and familiarity; gamers would

become acclimatized to such musical ma-nipulations, and their potency as triggers forphysiological responses would be diluted.It is also unlikely that the games industrywould wish to introduce creative guidelinesthat could result in samey musical outputs.


16/22

: Game Biosensors

Te current lack of standardization withinthe games industry, and a desire to producegames exhibiting originality and innovation,potentially predetermines against the adop-

tion of such an approach. Even if satisfactory musical solutionscould be found, challenges and questionswould still remain. For example, if biosensordata becomes the focus of interactivity ingames, such that the involuntary physiologi-cal and subconscious responses of playersgovern the dynamic and adaptive changesoccurring during game-play, might players

begin to lose their sense of conscious andvoluntary control when gaming? Perhaps game developers might preservethe players sensation of control against anadaptive game environment that respondsto changing physiological data, by creating ahierarchical prioritization structure withinthe programming of the game, such that vol-untary responses take precedence when mul-

tiple interactive data streams are detected.Whatever the potential solutions to this chal-lenge might be, other significant questionswould still need to be addressed. For example, evidence already outlinedshows that familiarity with musical stimuliresults in changes to physiological response,such as heart rate. Familiarity, repetition,and predictability are already problematic

issues for game developers seeking to creategames capable of sustaining the interest ofplayers for many hours. Te fact that physi-ological responses change as players becomefamiliarized with game elements, such asmusic and sound, adds a further layer ofcomplication when developing games relyingon biosensor data of this sort. In addition, familiarity with certain musi-

cal and sound-based stimuli might not beacquired as a result of gaming alone. Forexample, when playing Dead Spaceand en-countering its music for the first time, thephysiological responses of someone with apenchant for avant-garde music (in its less

approachable forms) would probably differfrom those of someone who is completelyunfamiliar with music of this type. Any as-sumptions that game developers make re-

garding likely physiological responses mightbe undermined by exceptions such as this,potentially creating gaming experiences thatare not satisfactory for all. Tis raises the question of standardization;something that already presents a challengeto the games industry. Physiological re-sponses are not standardized or universal forall. At most, scientific studies reveal general

patterns, indicating likely types of responseand associated parameters, which might bedescribed as normal or within the normalrange. However, other factors can result inexceptions. Familiarity has already beenconsidered, but physical and mental healthconditions might also alter physiologicalresponses to musical stimuli, as outlinedearlier in relation to brain injury patients.

Variables such as these would make it verydifficult for the games industry to developstandardized systems capable of universallypredicting, measuring, and integrating phys-iological responses. Tere might also be additional medicaland ethical issues to consider in the develop-ment of games utilizing biosensor data. Forexample, a biosensor detecting an increase in

heart rate might trigger adaptations withinthe game designed to further increase the in-tensity of this response. If the player however,suffers from a heart condition, this outcomemight carry a significant risk. Te use of bio-sensors to measure physiological responses,and the use of this data to sustain, intensify,and manipulate those responses through ad-aptations within the game, connect the player

far more directly to the gaming experience. As technologies develop with the capabilityof enveloping players in gaming experiencesthat feel more physical and vivid, the bound-ary between real experiences and virtual gameexperiences might become more blurred. For


17/22

. /

games carrying higher age certifications, thismight be a cause of ethical concern. It certainly seems likely that as new im-mersive technologies evolve alongside bio-

metric gaming and are used in conjunctionwith them, the sense of being ingame worldswill be heightened still further. Te founderof the NeuroGaming Conference, ZachLynch, outlined this very scenario in a recentinterview:

I think the answer is convergence. Its Oc-ulusRi[]with a Foc.usheadset

[]to im-prove your concentration with an EEG to

understand your emotional reactions, tofacial tracking technology to capture pupildilation and figure out what youre actu-ally looking at. All four of those combinedwith sound, some haptics[]orientedheadsets, in a haptics oriented chair.

Exciting though such developments mayappear to enthusiastic gamers, futurists,and game industry insiders, it is undeniablethat the convergence of such technologiesonly serves to magnify ethical and healthconcerns relating to biometric gaming, par-ticularly with regard to higher certificationgames. Such ethical and health concernscannot simply be addressed by the certifica-tion of games alone as a catch all means ofpreventing harm. At some point the industrywill need to address these questions withspecific reference to rapidly evolving devel-opments in biometric and immersive gamingtechnologies. Otherwise, the intensificationor reinforcement of physiological responsesduring game-play might result in undesir-able psychological, physical, or behavioralimpacts.

The Benefits

Te development of game technologies de-signed to capture and react to physiologicalresponses clearly presents new challenges,only some of which have been outlinedabove. Aside from any potential difficulties

or concerns presented by these new tech-nologies, it would be wrong to suggest thatfuture developments in this area might notresult in wider benefits.

Within the context of music therapy, forexample, the development of user-friendlytechnologies, which measure physiologicalreactions and then respond by triggeringmusical events designed to prescriptively treatthrough the generation of positive physiologi-cal responses, might provide an additionaland useful tool for music therapists. Further health benefits might be found in

the treatment of significant medical prob-lems such as dementia. Scientific evidencealready exists that proves that music elicitsassociations in the memory and can act as acue to emotional memory.Academic andscientific research exploring the potential ofmusic as a means of treating dementia andrelated conditions is ongoing, but there issignificant interest in developing technolo-

gies that use music as a form of prescriptivetreatment wherein the music adapts to par-ticular brain responses. Alternatively, such technologies might beapplied as simple forms of stress relief or aspick-me-ups, either through recreationalgaming, inducing positive physiologicalresponses, or through related, nongamingspin-off products, simply designed to de-

stress or li the spirits when relaxing.

Conclusion

Te games industry appears to be standingon the brink of a new phase in its develop-ment, one in which the human body itselfbecomes the focus of interface development.Te development of biosensors designed tocapture the diverse range of complex physi-

ological responses produced by the humanbody is already in progress. One of the pri-mary triggers for such responses in gamingis music and sound. Recent years have pro-duced a growing body of scientific research


18/22

: Game Biosensors

that indicates that particular types of musicand sound are capable of triggering signifi-cant and measurable physiological reactions.Te technical and creative challenges for

games developers seeking to exploit thesefindings within the context of gaming aresignificant. However, the commercial suc-cess of games such as Dead Space , with itsproven ability to trigger strong physiologicalresponses, would indicate that the ability totrigger significant physiological responsescarries a certain financial incentive. Moti-vated by the potential to outstrip competitors

in a crowded marketplace through new de-velopments in gaming and associated hard-ware, it seems likely that these challengeswill be addressed to some degree. In the case of games designed to inducenegative physiological responses, the suc-cessful achievement of these goals potentiallyraises some legitimate concerns relating tohealth, ethics, and even behavior. However

it is also true that in its quest to create theultimate gaming experience, the games in-dustry might develop technologies with thepotential to unlock the therapeutic power ofmusic through adapted applications in musictherapy, medicine, and the home. If so, thisnext phase of development in the industrymight offer new avenues of exploration andrelated technological developments with the

potential to tap the curative benefits of musicfor gamers and nongamers alike.

. Electrodermal or galvanic response (the abil-ity of the skin to conduct electricity) is used tomeasure emotional or physiological arousal. Tereare different types of electrodermal measurementused in research today; for a summary, see http://www.bem.fi/book//.htm, accessed August ,

. For more detailed information regarding theuse of skin conductance in research of this type,see Bernd Figner and Ryan O. Murphy, UsingSkin Conductance in Judgment and DecisionMaking Research, inA Handbook of Process rac-ing Methods for Decision Research, ed. Michael

Schulte-Mecklenbeck et al. (New York: Psychol-ogy Press, ), . . Further details available at http://www.foc.us/,accessed August , . . See http://www.indiegogo.com/projects/

muse-the-brain-sensing-headband-that-lets-you-control-things-with-your-mind, accessed August, . . Electroencephalogrammeasures the electri-cal activity of the brain. . ests conducted by the company VerticalSlice. See http://www.gamasutra.com/view/feature//scary_game_findings_a_study_of_.php?print=, accessed August , . . aken from a telephone interview with theauthor (). . Te second game in the series makes greateruse of thematic material than its predecessor. . For Dead Space, Jason Graves composed alarge number of stingers (surprise hits or cues),which he called his bucket of fear. Tese stingerscan be triggered at random when required by thegame engine and integrated dynamically withinthe music mix.

. See Eduardo Coutinho and Angelo Cange-losi, Musical Emotions: Predicting Second-by-Second Subjective Feelings of Emotion from Low-Level Psychoacoustic Features and PhysiologicalMeasurements, Emotion, no. (): . . As Coutinho and Cangelosi suggest, thegoal-oriented nature usually attributed to emo-tions clashes with musics ability to elicit all kindsof strong and mild emotional reactions in its lis-teners, an aspect that has been at the very center

of a long-standing controversial discussion: Canmusic induce emotions? Are MEs just like other(real) emotions? Ibid., . . Gosselin et al., Impaired Recognition ofScary Music following Unilateral emporal LobeExcision, Brain, no. (): . In thisstudy, pieces were rated on a dissonant scale from to . . Nathalie Gosselin, Sverine Samson, RalphAdolphs, Marion Noulhiane, Mathieu Roy,

Dominique Hasboun, Michel Baulac, and IsabellePeretz, Emotional Responses to UnpleasantMusic Correlates with Damage to the Parahip-pocampal Cortex, Brain, no. (): -. In this study, Western Classical instrumentalexcerpts were altered by transposing the melody


19/22

. /

lines up or down a semitone leaving the accom-paniments unaltered, thus creating dissonance. Anumber of other studies have applied similar ap-proaches in the creation of dissonant stimuli. SeeStefan Koelsch, Tomas Fritz, Yves V. Cramon,

Karsten Mller, and Angela D. Friederici., In-vestigating Emotion with Music: An fMRI Study,Human Brain Mapping (): ; andDaniela Sammler, Maren Grigutsch, Tomas Fritz,and Stefan Koelsch, Music and Emotion: Elec-trophysiological Correlates of the Processing ofPleasant and Unpleasant Music, Psychophysiology, no. (): . In both of these studies,instrumental recordings of dance tunes from thelast four centuries were electronically processed,simultaneously combining the original recordingswith two pitch shied versions (one a tone aboveand another a tritone below). . Enrique O. Flores-Gutirrez, Jos-Lus Daz,Fernando A. Barrios, Rafael Favila-Humara,Miguel . Guevara, Yolanda del Ro-Portilla, andMaria Corsi-Cabrera, Metabolic and ElectricBrain Patterns During Pleasant and UnpleasantEmotions Induced by Music Masterpieces,In-

ternational Journal of Psychophysiology ():. Tis study used atonal music from the filmscore Danton, composed by J. Prodromids as astimulus for fear. . Sammler et al., Music and Emotion, .Also Koelsch et al., Investigating Emotion . . .fMRI Study, . . Valence being a scale of response rangingfrom positive to negative (good/ bad or pleasure/displeasure); arousal representing a range from

calm and peaceful to excited and energetic. . As in Flores-Gutirrez et al., Metabolic andElectric Brain Patterns, . . Brain response, heart rate, and skin conduc-tance. Respiration is not included in this study. . Te amygdala is a key component of thelimbic system in the brain and is involved in theexperience of anxiety, distress, and fear. One studydescribes it as the heart of the brains emotionalsystem and states that the structure is involved

in simple fear conditioning, contextual condi-tioning, recognition of fearful facial expressions,and emotion-guided decision making (EranEldar, Ori Ganor, Roee Admon, Avraham Bleich,alma Hendler, Feeling the Real World: LimbicResponse to Music Depends on Related Content,

Cerebral Cortex , no. (): .) Teamygdala has also been found to be involved inthe processing of positive emotions (ElisabethA. Murray E, Te Amygdala, Reward and Emo-tion, rends in Cognitive Sciences, no. []:

). In a review article, Stefan Koelschsuggests that different nuclei of the amygdalaare involved in modulating activity of differentemotion networks. See owards a Neural Basisof Music-Evoked Emotions, rends in CognitiveSciences, no. (): . . For further details, see Joseph LeDoux, TeEmotional Brain, Fear, and the Amygdala, Cellu-lar and Molecular Neurobiology, nos. / ():. See also J. LeDoux, Rethinking theEmotional Brain, Neuron (February , ):. . Ibid., . . Gosselin et al., Emotional Responses to Un-pleasant Music, . Te authors cite a numberof studies, all of which appear to confirm the asso-ciation between dissonance and unpleasantness. . Koelsch et al., Investigating Emotion . . .fMRI Study, . Dissonant musical stimuli

electronically pitch processed melody (originalmelody combined with two pitch shied versionsone up a tone, one down a tritone). Mean duration seconds. . Te Hippocampus functions as an importantcomponent of the limbic system and of memoryprocessing. . Te parahippocampal gyrus receives directprojections from visuospatial processing areas. . . and substantial input from other polymodal

areas and has reciprocal connections with [otherstructures in the brain such as] . . . the amygdala,hippocampus and striatum in mediating distinctforms of memory (http://www.bookrags.com/tandf/parahippocampal-gyrus-tf/, accessed Au-gust , ). . Te most prominent anterior part of thetemporal lobes of the brain. . Anne J. Blood, Robert J. Zatorre, Patrick Ber-mudez, and Alan C. Evans., Emotional Responses

to Pleasant and Unpleasant Music Correlate withActivity in Paralimbic Brain Regions, Nature Neu-roscience, no. (): ; Blood and Zatorre,Intensely Pleasurable Responses to Music Cor-relate with Activity in Brain Regions Implicated inReward and Emotion, Proceedings of the National


20/22

: Game Biosensors

Academy of Sciences of the United States of America, no. (): ; Nathalie Gosselin,Isabelle Peretz, Marion Noulhiane, DominiqueHasboun, Christine Beckett, Michel Baulac, andSverine Samson, Impaired Recognition of Scary

Music following Unilateral emporal Lobe Exci-sion, Brain, no. (): ; Gosselin etal., Emotional Responses, . . Gosselin et al., Impaired Recognition,. . Tere is some debate about the functionsof the parahippocampal cortex (PHC). It hasbeen implicated in the processing of place-re-lated information. It has also been implicated inepisodic memory. Others suggest that the PHCplays a role in processing contextual associationsin general (http://cercor.oxfordjournals.org/content///.abstract, accessed August ,). Others argue that it is concerned with therepresentation of three dimensional local spaceand objects (see http://www.ncbi.nlm.nih.gov/pubmed/, accessed August , ). . Gosselin et al., Emotional Responses,.

. Koelsch, owards a Neural Basis, . . See endnotes and . . Stimuli presented in consecutive secondsegments, alternated with seconds of noise, fora total duration of minutes. . Flores-Gutirrez et al., Metabolic andElectric Brain Patterns, . For previous findingsrelating frontal activation to positive or negativeaffect, see Richard J. Davidson, Anterior CerebralAsymmetry and the Nature of Emotion, Brain

and Cognition, no. (): . . Ibid., . . LeDoux, Rethinking the EmotionalBrain,. . Ibid., . . Consisting of the brain and spinal cord. . Te part of the vertebrate nervous systemthat regulates involuntary action, as of the intes-tines, heart, and glands, and that is divided into thesympathetic nervous system and the parasympa-

thetic nervous system (http://www.thefreedictionary.com/autonomic+nervous+system, ac-cessed August , ). . LeDoux, Rethinking the Emotional Brain,, provides further citations of a number ofthese studies.

. However, as Kreibig points out, positionson the degree of specificity of ANS activation inemotion . . . greatly diverge. Sylvia D. Kreibig,Autonomic Nervous System Activity in Emotion:A Review, Biological Psychology (): .

. However, as one study points out, thesestudies do not exclude that cultural experiencescan modify judgments about the pleasantness ofcertain dissonances. Koelsch et al., InvestigatingEmotion . . . fMRI Study, . . For citations of some of these studies, seeCoutinho and Cangelosi, Musical Emotions, . . Ibid. . Kreibig, Autonomic Nervous System Activ-ity in Emotion, . . Coutinho and Cangelosi, Musical Emo-tions, . . Julian F. Tayer and Robert W. Levenson,Effects of Music on Psychophysiological Re-sponses to Stressful Film, Psychomusicology,no. (): . Tis study measured the slowvarying component of skin conductance, definedas skin conductance level or SCL. . Margaret M. Bradley and Peter J. Lang,

Affective Reactions to Acoustic Stimuli, Psycho-physiology, no. (): ; Sammler et al.,Music and Emotion, ; Robert J. Ellis andRobert F. Simons, Te Impact of Music Subjec-tive and Physiological Indices of Emotion WhileViewing Films, Psychomusicology, no. ():. (Tis study measured the faster changingcomponent of skin conductance, known as skinconductance response or SCR. SCR is associatedwith the activity of the palmar sweat gland.)

. Kreibig suggests that these results indicatean element of passivity, and may be taken to sug-gest vagal mediation. Kreibig, Autonomic Ner-vous System Activity in Emotion, . . Coutinho and Cangelosi, Musical Emo-tions, . . Based on findings of a previous study thatfound evidence that spatiotemporal dynamics inpsychoacoustic features resonate with two psy-chological dimensions of affect underlying judg-

ments of subjective feelings: arousal and valence.Coutinho and Nicola Dibben, PsychoacousticCues to Emotion in Speech Prosody and Music,Cognition and Emotion, no. (): . . Coutinho and Cangelosi, Musical Emo-tions, .


21/22

. /

. Bradley and Lang, Affective Reactions, . . Ibid. . Ibid., . . Hannah Lynch, Parth Patel, Stanislav Kon-rath, and Devin Blodgett, Te Effect of Music as

a Prepulse Stimulus on the Activation of the Sym-pathetic Nervous System (), , http://jass.neuro.wisc.edu///LabGroupTeEffectofMusic.pdf, ac-cessed August , . . Janet E. Landreth and Hobart F. Landreth,Effects of Music on Physiological Response,Journal of Research in Music Education, no. (): . . Margaret M. Bradley, Natural SelectiveAttention: Orienting and Emotion, Psychophysiol-ogy, no. (January ): . . Ibid. Bradley cites a number of studies ex-ploring responses to novel and significant stimuli . Landreth and Landreth, Effects of Music, . . While there is some evidence suggesting thatthere might be an additive relationship when musi-cal and film stimuli are presented simultaneously, itis thought that this additive affect is less significant

when the visual stimuli is highly negative (Ellis andSimons, Te Impact of Music, ). . Markellos Nomikos, Edward Opton Jr.,James Averill, and Richard S. Lazarus, SurpriseVersus Suspense in the Production of Stress Reac-tion,Journal of Personality and Social Psychology, no. , pt. (): . Tis study does notspecify which type of skin conductance measurewas adopted. . Stefan Koelsch, Investigating Emotion with

Music: Neuroscientific Approaches,Annals ofthe New York Academy of Sciences ():; Koelsch et al., Investigating Emotion . . .fMRI Study, ; Sammler et al., Music andEmotion, ; Emmanuel Bigand, Richard Parn-cutt, and Fred Lerdahl, Perception of Musicalension in Short Chord Sequences: Te Influenceof Harmonic Function, Sensory Dissonance,Horizontal Motion, and Musical raining, Per-ception and Psychophysics, no. (): ;

Carol Krumhansl, An Exploratory Study of Mu-sical Emotions and Psychophysiology, CanadianJournal of Experimental Psychology, no. ():. (Te Krumhansl study used skin con-ductance levels as one measure of physiologicalresponse, amongst others).

. Sammler et al., Music and Emotion, . . Koelsch, Investigating Emotion . . . Neuro-scientific Approaches, . Teodor Adorno and Hanns Eisler, Compos-ing for the Films(London: Continuum, ), .

(Te first edition was published in ). . See Lynch et al.,Te Effect of Music as aPrepulse Stimulus, . . Bigand et al., Perception of Musical en-sion, . . Koelsch, Investigating Emotion . . . Neuro-scientific Approaches, . . Nikolaus Steinbeis, Stefan Koelsch, and JohnA. Slobodab, Emotional Processing of HarmonicExpectancy Violations,Annals of the New YorkAcademy of Sciences (): . . Bigand et al., Perception of Musical en-sion, . . Arne hman, Preattentive Processes in theGeneration of Emotions, in Cognitive Perspectiveson Emotion and Motivation, ed. Vernon Hamiltonet al. (Dordrecht: Kluwer Academic, ), . . Klaus Scherer, Which Emotions can beInduced by Music? What are the Underlying

Mechanisms? And How can we Measure Tem?,Journal of New Music Research, no. (): . . Laurel J. rainor, Christine D. sang, andVivian H. W. Cheung, Preference for SensoryConsonance in - and -Month-Old Infants,Music Perception, no. (): . Laurel J. rainor and Becky M. Heinmiller,Te Development of Evaluative Responses toMusic: Infants Prefer to Listen to Consonanceover Dissonance, Infant Behavior and Develop-

ment, no. (): . . Marcel R. Zentner and Jerome Kagan. In-fants Perception of Consonance and Dissonancein Music, Infant Behavior and Development(): . . rainor and Heinmiller, Development ofEvaluative Responses, . . Ibid., . . Glenn E. Schellenberg and Laurel J. rainor,Sensory Consonance and the Perceptual Similar-

ity of Complex-one Harmonic Intervals: ests ofAdult and Infant Listeners,Journal of the Acousti-cal Society of America, no. (): ;Glenn E. Schellenberg and Sandra E. rehub,Natural Musical Intervals: Evidence from InfantListeners, Psychological Science, no. ():


22/22

: Game Biosensors

; Laurel J. rainor, Te Effect of FrequencyRatio on Infants and Adults Discrimination ofSimultaneous Intervals,Journal of ExperimentalPsychology: Human Perception and Performance, no. (): .

. rainor and Heinmiller, Development ofEvaluative Responses, . . Ibid. . Ibid. . Gosselin et al., Impaired Recognition, . . Patrik N. Juslin and Petri Laukka, Com-munication of Emotions in Vocal Expression andMusic Performance: Different Channels, SameCode? Psychology Bulletin, no. (): . A study by Coutinho and Dibben suggeststhat with regards to speech prosody and music,the subset of psychoacoustic cues that carry emo-tional meaning is very similar. PsychoacousticCues to Emotion, . . Gosselin et al., Impaired Recognition, . . Berthoid Hoeckner, Emma W. Wyatt, JeanDecety, and Howard Nusbaum, Film Music Influ-ences How Viewers Relate to Movie Characters,Psychology of Aesthetics, Creativity, and the Arts,

no. (): . . Kreibig, Autonomic Nervous System Activ-ity in Emotion, . . Industry-specific soware (such as Audioki-netics Wwise and Firelight echnologies Fmod)developed as audio toolkits for composers andsound designers, to facilitate integration withindynamic mixes. . Coutinho and Cangelosi, Musical Emo-tions, ; Coutinho and Dibben, Psycho-

acoustic Cues to Emotion, . See HeartRate and Skin Conductance, above, for an outlineof predictive computer models. . Coutinho and Cangelosi, Musical Emo-tions, . . A virtual reality headset for immersivegaming. See http://www.oculusvr.com/, accessedAugust , . . A headset for gamers, which delivers tran-scranial Direct Current Stimulation (tDCS). See

http://www.foc.us/, accessed August , . . Haptic technologies can be used in manydifferent applications, and integrated into manydifferent physical objects, from hand-held devices

through to furniture, shoes, vests, and gloves. Teaim of the technology is to ensure that what isseen is also felt, thereby creating further sensoryfeedback. . See http://neurogadget.com////

interview-with-neurogaming-conference-founder-zack-lynch/more-, accessed August ,. . Klaus Scherer, Which Emotions can be In-duced by Music?, . See also Stefan Koelsch,A Neuroscientific Perspective on Music Terapy,Te Neurosciences and Music IIIDisorders and

Plasticity: Ann. N.Y. Acad. Sci. (): .

Baumgartner, Tomas, Kai Lutz, Conny Schmidtand Lutz Jncke. Te Emotional Power ofMusic: How Music Enhances the Feeling ofAffective Pictures. Brain Research, no. (): .

Dalgleish, im. Te Emotional Brain. Na-ture Reviews Neuroscience (): .doi:./nrn.

Lynch, Michael P., Rebecca E. Eilers, D. Kim-

brough Oller, and Richard C. Urbano. In-nateness, Experience, and Music Perception.Psychological Science, no. (): . doi:./j.-..tb.x.

Meyer, Leonard. B. Emotion and Meaning inMusic. Chicago: University of Chicago Press,.

Peretti, Peter O., and Kathy Swenson. Effects ofMusic on Anxiety as Determined by Physi-ological Skin Responses.Journal of Research in

Music Education, no. (): .rainor, Laurel J., and Sandra E. rehub. A

Comparison of Infants and Adults Sensitivityto Western Musical Structure.Journal of Ex-perimental Psychology: Human Perception and

Performance, no. (): .Ulrich, Roger S., Robert F. Simons, Barbara D.

Losito, Evelyn Fiorito, Mark A. Miles, and Mi-chael Zelson. Stress Recovery during Exposureto Natural and Urban Environments.Journal of

Environmental Psychology(): .Zentner, Marcel R., and Jerome Kagan. Percep-

tion of Music by Infants. Nature, no.

(): .

Plug-In to Fear - Game Biosensors and Negative Physiological Responses to Music 8.1.Mitchell

Documents

Transcript of Plug-In to Fear - Game Biosensors and Negative Physiological Responses to Music 8.1.Mitchell