PLEASE SCROLL DOWN FOR ARTICLE - h. Wmyl/Papers/AAIPublished.pdf · PLEASE SCROLL DOWN FOR ARTICLE...

PLEASE SCROLL DOWN FOR ARTICLE

This article was downloaded by: [Lim, Mei Yii]On: 16 October 2009Access details: Access Details: [subscription number 915926948]Publisher Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Applied Artificial IntelligencePublication details, including instructions for authors and subscription information:http://www.informaworld.com/smpp/title~content=t713191765

AN EMERGENT EMOTION MODEL FOR AN AFFECTIVE MOBILE GUIDEWITH ATTITUDEMei Yii Lim a; Ruth Aylett a

a School of Mathematical and Computer Sciences, Heriot Watt University, Edinburgh, Scotland, UK

Online Publication Date: 01 October 2009

To cite this Article Lim, Mei Yii and Aylett, Ruth(2009)'AN EMERGENT EMOTION MODEL FOR AN AFFECTIVE MOBILE GUIDEWITH ATTITUDE',Applied Artificial Intelligence,23:9,835 — 854

To link to this Article: DOI: 10.1080/08839510903246518

URL: http://dx.doi.org/10.1080/08839510903246518

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf

This article may be used for research, teaching and private study purposes. Any substantial orsystematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply ordistribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae and drug dosesshould be independently verified with primary sources. The publisher shall not be liable for any loss,actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directlyor indirectly in connection with or arising out of the use of this material.

http://www.informaworld.com/smpp/title~content=t713191765

http://dx.doi.org/10.1080/08839510903246518

http://www.informaworld.com/terms-and-conditions-of-access.pdf

Applied Artificial Intelligence, 23:835–854Copyright © 2009 Taylor & Francis Group, LLCISSN: 0883-9514 print/1087-6545 onlineDOI: 10.1080/08839510903246518

AN EMERGENT EMOTION MODEL FOR AN AFFECTIVEMOBILE GUIDE WITH ATTITUDE

Mei Yii Lim and Ruth AylettSchool of Mathematical and Computer Sciences, Heriot Watt University,Edinburgh, Scotland, UK

� This article describes a novel emergent emotion model for a context-aware “Affective MobileGuide with Attitude,” a guide with emotions and personality that accompanies visitors touringan outdoor attraction. The interest lies in modelling the conditions to the emergence of emotionsinstead of prescripting emotions. The emotion model regulates the guide’s emotions, behaviorand beliefs, allowing the guide to adapt to its interaction environment flexibly. The guideadopts improvisational storytelling techniques taking into consideration the user’s interests, itsown interests, and its current memory activation in narration. It has emotional memory thatstores its previous experiences allowing it to express its perspectives while narrating the stories.The guide’s internal state is expressed through a 2-dimensional face where different facialfeatures vary from one state to another along the arousal-pleasure emotional space. The guideexhibits potential in improving overall tour experience verified by a user evaluation.

The idea of improving tour assistance by agents capable of respondingto the needs and emotional states of the users is not new, yet not muchhas been achieved so far. Users often lose their interest while interactingwith current virtual guides due to unmet expectations about the characterintelligence and a lack of “life.” One way to tackle this problem is toprovide the agent with an emotional system inspired by the human’s sothat it can generate behavior that produces an “illusion of life.”

This article reports on a novel-emergent emotion model aiming at thecreation of a biologically plausible “Affective Mobile Guide with Attitude,”hereafter termed as the Affective Guide. This work is part of the resultof a doctoral thesis (Lim 2007). What we mean by “attitude” is that

The authors would like to thank the EU FP6 Network of Excellence HUMAINE (Human-Machine Interaction Network on Emotion, http://emotion-research.net) for funding this work.

Address correspondence to Mei Yii Lim, School of Mathematical and Computer Sciences,Heriot Watt University, Edinburgh, Scotland EH14 4AS, UK. E-mail: [email protected]

Downloaded By: [Lim, Mei Yii] At: 16:16 16 October 2009

836 M. Y. Lim and R. Aylett

the guide is capable of presenting stories from its own perspective. Theresulting system provides guidance, interesting and engaging interactionaround outdoor attractions on a mobile platform. The Affective Guidecommences the tour by introducing itself. This is followed by an ice-breaking session where it extracts information about the user’s nameand interests. Then, it chooses attractions that match the user’s interests,and plans the shortest possible route to these destinations. After a briefintroduction to the tour, it navigates the user to the chosen locations bygiving directional instructions. Upon arrival at a planned destination, itnotifies the user and starts the storytelling process. We move away froma guide that recites facts to a guide that tells stories by improvisation.The guide tells facts as well as its autobiography by linking its memories—semantic and emotional—its own interests, and the user’s interests to thephysical location, thereby augmenting the real world with an additionallayer of virtual information. It continuously receives input from the userand shapes the stories according to his/her interests. The user inputsfeedback through a graphical user interface (GUI) and receives output bymeans of text, speech, and visual animation.

We start with a review of work that has inspired our research. Followingthis, we present a brief description of the Affective Guide. The coreelement of this research, the biologically inspired emotion model for anAffective Guide is then exposed, featuring the integration of an emotionmodel, a storytelling system, emotional memories, and a facial animationsystem. Since the storytelling system and emotional memories (Lim andAylett 2007b) and the facial animation system (Lim and Aylett 2007c) havebeen published elsewhere, we will describe these components only brieflyand focus on detailing how the emotion model drives these components.The algorithms involved are documented. Reports on evaluations with realusers to establish the impact of the inclusion of emotions and attitude inthe guide can be found in Lim and Aylett (2007a) and Lim (2007).

INSPIRATION FOR OUR WORK

Much work has been done to create context-aware applications thatpersonalize information presentation. Some of these systems are mobilewhile others are virtual. Some examples of mobile context-aware tourguide systems are C-MAP (Sumi et al. 1998), HIPS (O’Grady, O’Rafferty,and O’Hare 1999), GUIDE (Cheverst et al. 2000), HyperAudio (Petrelliand Not 2005), and PEACH (Stock and Zancanaro 2007). All these systemsutilize multimodality to overcome the static constraints of the environmentby dynamically changing the user’s perception of the environment throughthe use of multimedia techniques and changing the user’s physical locationthrough suggestions of where to go next. While HIPS and HyperAudiodo not include life-like interface character, C-MAP, GUIDE, and PEACH


Emergent Emotion Model for an Affective Mobile Guide 837

do. Other examples of systems that employ life-like interface character arevirtual tour guide systems such as Kyoto Tour Guide (Isbister and Doyle1999), the system by Almeida and Yokoi (2003), and the SAGRES VirtualMuseum (Bertolleti, Moraes, and da Rocha Costa 2001). Evaluations ofthese systems reveal that the characters’ human-like behavior offers usersa more friendly interaction interface and increases the suspension ofdisbelief in users, at the same time helping them to learn and encouragethem to explore for more information.

According to Nass, Steuer, and Tauber (1994), the individual’sinteraction with computers is inherently natural and social. Becauseaffective communication occurs naturally between people, it is expected bypeople when they interact with computers. Picard (1997) argued that forcomputers to be truly effective at decision-making, they will have to haveemotion-like mechanisms working in concert with their rule-based systems.Hence, a guide that shows variable emotions, attitude, and adaptivebehavior is likely to be perceived as more intelligent and believable.We view the human emotional system to be consisting of a successfulbody-mind link (Damasio 1994) where physiological responses influencecognitive processing and vice versa, resulting in a continuous feedbackloop between the two levels. Thus, we based our emotion architectureon Dörner’s model (Dörner 2003; Dörner and Hille 1995), “Psi,” thatintegrates motivation (physiological drives), emotion, and cognition forhuman action regulation. Additionally, we aim to develop a biologicallyplausible agent and the “Psi” model has successfully replicated humanbehavior in complex tasks (Bartl and Dörner 1998; Dörner 2003; Dörner,Gerdes, Mayer, and Misra 2006).

Motivation in “Psi” represents needs that influence survival and canbe existential (e.g., need for water and affiliation) or intellectual (e.g.,need for competence and certainty). Emotions are not explicitly definedbut emerge from modulations of cognitive processes while cognition refersto those processes that control adaptive behaviors, including perception,action-selection, planning, and memory access. Modulating parameters ofcognitive processes that define emotions are:

• Arousal: Speed of information processing• Resolution level: Carefulness or attentiveness of behavior• Selection threshold: Tendency to concentrate on a particular intention.

Complex behaviors become apparent when the modulatingparameters’ values are modified by needs resulting in what can betermed as emotions. For example, fear is experienced under conditionsthat produce high needs, and as a result, is characterized by a higharousal (quick reaction), low-resolution level (inaccurate perception andplanning), and low selection threshold (easily distracted by other cues



within the environment in an attempt to detect dangers). The interactionbetween motivation, emotion, and cognition is a continuous loop withinthe organism, regulated by both internal and external circumstances.Whenever a need deviates from the set point, it activates the correspondingmotives. Several motives may be active at a particular time and one ofthese motives is selected according to an expectancy-value principle. Theselected motive is the actual intention that is executed. There are threedifferent modes of execution: automatism, knowledge-based, or trial-and-error. If the “Psi” agent is highly experienced in performing the currentintention, a complete course of action to achieve the goal is carried outautomatically; otherwise it generates a plan to achieve the goal. If boththese fail, it explores the environment to collect information that will leadto the goal. Additionally, it learns by experience and possesses a memorysystem in which all perceptions and activities are continuously recorded.

Many other interesting emotion models exist, but these modelsconcentrate either on the physiological aspects (Cañamero 1997; Pezzuloand Calvi 2007; Velásquez 1998) or the cognitive aspects (Gratch andMarsella 2004; Ortony, Clore, and Collins 1988; Turrini, Meyer, andCastelfranchi 2007) of emotions, lacking a communication bridge betweenthe two aspects. The physiological models operate at a nonsymboliclevel which is too low for the Affective Guide framework. The guiderequires cognitive capabilities for planning and storytelling. In contrast,the cognitive models based on appraisal theories perceive emotions asarising from a certain kind of cognition, with little or no attentionto the other important aspects of emotion such as the physiological,behavioral, and expressive components. Additionally, we adopt the “affectas interaction” view (Boehner, DePaula, Dourish, and Sengers 2005), thusruling out models that define emotions explicitly. Operating in a real-worldenvironment means having to deal with many uncertain circumstances;hence, an architecture that is flexible and robust such as “Psi” is necessaryto allow the agent to cope with uncertainty, react to unanticipated events,and recover dynamically in the case of poor decisions.

For storytelling, we must acknowledge the work of Ibanez (2004), inparticular its emphasis on improvisational storytelling. The work focusedon improvisational virtual storytellers in a virtual environment, hence, lacksthe user’s interaction. While Ibanez’s system omits user interests duringnarration, we consider this together with the user’s feedback throughoutthe tour session as important factors that may affect the overall tourexperience. The Affective Guide construct stories based on historical facts,its past experiences, and the user’s interests.

Facial behavior has been commonly categorized using two judgmentprocedures: categorical and dimensional. A particular emotion can usuallybe represented by several facial expressions, depending on intensity,



and the discrete approach suffers from the flaw of rigidity, with its one-to-one mapping. The dimensional approach eliminates this restriction,conveying a wider range of affective messages. Using the dimensionalapproach, the only consistent finding across experiments for classifyingfacial behavior is the arousal-pleasure dimensions (Ekman and Friesen1982) leading to its application in mapping facial expressions toemotions (e.g., Russell (1997) and Grammer and Oberzaucher (2006)).The dimensional approach supports the Affective Guide’s underlyingarchitecture for emerging emotional states. However, the mappingadopted in the Affective Guide differs from previous works in that itmapped different facial features (eyes, mouth, and eyebrows) rather thanfacial action units (Ekman and Friesen 1982) onto the arousal-pleasuredimensions of emotion.

THE AFFECTIVE GUIDE

The Affective Guide integrates a PDA with a global positioning systemand an embedded text-to-speech system. Due to limited resources on thePDA, a server is utilized to hold the data, perform processing, and transmitinformation to the hand-held unit on demand. Communication betweenthe PDA and the server is through wireless LAN as depicted in Figure 1.

Figure 2 presents the main interaction interface during the toursession. The text is displayed on the screen allowing the user to read anyinformation that may have been missed when listening to the speech. Theuser has the freedom to stop, pause, resume, or repeat the presentationwhenever they wish. The guide has abilities to detect the user location,replan the route when the user strays from the planned path, generate andpresent stories automatically. Notification of current location using bothspeech and message boxes are provided periodically to ensure continuityin system behavior.

FIGURE 1 The overall system architecture.



FIGURE 2 The main graphical user interface.

On the way to the destination, if the user is attracted to a site which isnot in the preplanned route, he/she can activate the storytelling processmanually using the “Tell” button. This facility gives the user freedom towander around the attraction site at will during and after the plannedtour. At any time, the user can always find information about the functionof buttons from the “Help” menu. After each storytelling cycle, the usercan choose to listen to more stories about the current location or moveto the next location by touching the “More” or the “Continue” button,respectively. At this stage, the user has to provide feedback on his/herdegree of interest in the stories and degree of agreement with the guide’sargument(s). Each rating ranges from 0 to 10 on a sliding bar and reflectsthe user’s current feelings and opinions about the story content, useful forpersonalizing subsequent stories.

THE EMERGENT EMOTION MODEL

The Emergent Emotion Model is the novel element of this research.Taking the psychological model—“Psi” (Dörner 2003; Dörner and Hille1995) as a basis, we have added a storytelling system, emotional



memories, and a facial expression mechanism. We put more emphasison social interaction with humans and maintenance of cognitive needs,whereas Dörner’s work is more concerned with “survival” related issues.In our architecture, motivation is represented by the needs (built-inmotivators) of the guide to maintain its performance and establishan acceptable condition for interaction with the user. Emotions areemergently determined by the modulators while cognition is representedby the information processes in PERCEIVE, GENERATE INTENTION,SELECT INTENTION, RUN INTENTION, RECOVER, as well as in theMEMORY OF INTENTION represented by the dotted boxes in Figure 3.

Functionally, the guide perceives the environment continuously.It reads the user’s inputs which include the user’s degree of interest in thestories (DoI) and the user’s degree of agreement to the guide’s arguments(DoA); the system feedback, either success or failure in performing aselected intention; and the GPS information. Using this information,it updates its built-in motivator values and generates intention(s). Morethan one intention can be active at any point in time and the decisionon which and how to perform the intention is based on the guide’scurrent emotional state, influenced by the built-in motivators and themodulators. Finally, the execution of intention will produce a successor failure feedback into the system and recovery will be performed asnecessary.

FIGURE 3 The emergent emotion model.



When the interaction between the user and the guide starts, theguide’s long-term memories, both semantic and emotional, containa complete set of location-related information and the guide’s pastexperiences, respectively. In contrast, the user’s interests model is empty.The guide makes assumptions about the user’s interests based on the initialinformation from the ice-breaking session. As interaction proceeds, theguide constantly updates its user’s interest model so that narration is alwayspersonalized. The following discussion provides detailed descriptions foreach of these processes.

Built-in Motivators

In our context, the guide has two built-in motivators to maintain—competence and certainty. The level of competence refers to the guide’sability to narrate stories and cope with the user’s differing perspectiveabout an issue or event, whereas the level of certainty is the degreeof predictability of the environment (GPS reliability) and of the userinterests. For example, if the user disagrees with the guide’s opinion, itslevel of competence decreases due to lack of confidence in coping with thesituation. If the user likes the stories, its level of certainty increases becauseits prediction about the user’s interest is confirmed. Figure 4 illustrates theinteraction between the environment variables and the built-in motivatorsin the PERCEIVE process.

From Figure 4, it can be seen that two signals—positive andnegative—exist for each built-in motivator: efficiency versus inefficiencyfor competence and certainty versus uncertainty for certainty. Additionally,each built-in motivator has intensity and need values. Every time a newinput is received, all the attributes of each built-in motivator will beupdated. The value of the positive and negative signals are calculated as

FIGURE 4 Interaction between stimuli and the built-in motivators.



shown in Equations (1) and (2), respectively.

pSignal = (pSWeight)k∑

i=1

(fi)(iWeight) (1)

nSignal = (nSWeight)k∑

i=1

(1 − fi)(iWeight) (2)

In the equations, pSignal represents the positive signal (efficiency orcertainty) while nSignal represents the negative signal (inefficiency oruncertainty). pSWeight and nSWeight are the weights of the positive andnegative signals, respectively. fi denotes the environment variable values,which can be the GPS reliability, DoI, DoA and the system’s own feedback.k refers to the total number of variables that affect each built-in motivator’ssignals. iWeight refers to the degree of impact a particular variable hason the motivator. In the current version, both GPS reliability and DoIinfluence certainty but in the ratio of 1:3, which means the DoI has ahigher impact on the motivator than the GPS accuracy. This is becausethe feelings of the user are viewed as more important. The same ratioapplies for system success or failure and DoA in the case of competence.Thus, in the current system, the value of k is 2 because certainty anduncertainty are weighted sums over GPS reliability and DoI while efficiencyand inefficiency are weighted sums over system feedback and DoA. Usingthe values of the opposing signals, the intensity of the respective motivatorsis generated using Equation (3). This intensity value is maintained acrosseach cycle to ensure a gradual change in the guide’s internal state:

intensity = Max(0,Min(1�7, intensity + pSignal − nSignal)) (3)

need = 1 − (Log (1 + intensity)) (4)

Subsequently, calculations of needs are performed using Equation (4).A constant value of 1.7 (an approximation of e-1) is applied toEquation (3) to ensure that need values in Equation (4) range from 0 to 1.The need measures deviation from the set point indicator, referring to theneed for competence and the need for certainty. A need will tend toincrease in value unless it is satisfied. The guide’s aim is to keep thesevalues as low as possible. These need values will affect the modulatorvalues, discussed in the next section. The need values also determinethe state of the motivators, used for intention selection. Each built-inmotivator can be in one of the following five states at any time instant:LOW, MEDLOW (medium low), NEUTRAL, MEDHIGH (medium high),or HIGH depending on its urgency as shown in Figure 4. These states



form a fuzzy set that allows the guide to reactively select different copingstrategies under different need ranges to respond to its current interactioncircumstances.

Modulators

The guide has three modulating parameters of its emotional state:arousal level, resolution level, and selection threshold. Like the motivators,these modulators have an intensity that falls in the range of 0 and1 inclusive and a weight, a constant value that will shape the guide’spersonality, discussed in Lim, Aylett, and Jones (2005). The influence ofthe built-in motivators on the arousal level that further influences othermodulators is presented in Figure 5.

Arousal level refers to the speed of processing, that is, the guide’sreadiness to act. When arousal level is high, fast decisions and actions aresought. Thus, a high arousal tends to narrow attention and inhibit in-depthplanning. Its value (aIntensity) is calculated by summing the needs as shownin Equation (5). After the story generation process, if emotional storyelement(s) have been included, a final arousalIntensity that combines bothcurrent and memory arousal is calculated (Equation (6)). It is assumedthat the arousal strength of current experience is stronger compared to theretrieved emotional event; thus a weight of 0.4 is applied to the value frommemory while a weight of 0.6 is applied to the current arousal value. Thisassumption is made based on the fact that the guide is directly involvedin the currently happening event, hence experiencing a more intensefeedback from this event than the event it recalls from memory. If noemotional element has been included in the story, the final arousalIntensitytakes the value of the current aIntensity:

aIntensity = Log (1 + sumOfNeed)(weight) (5)

arousalIntensity = 0�4(memoryIntensity) + 0�6(aIntensity) (6)

FIGURE 5 Interaction between the built-in motivators and the modulators.



resolutionIntensity = (1 − arousal 2)(weight) (7)

selectionThreshold = (arousal 2)(weight) (8)

Arousal affects both the resolution level and the selection thresholdvalues. Resolution level (Equation (7)) covaries inversely while selectionthreshold (Equation (8)) covaries directly to the arousal level. Bothresolution level and selection threshold are distinguished from arousallevel so that the influence of weight parameters (in Equations (7)and (8)), which determine the guide’s personality, can be brought tobear. Resolution level determines the carefulness and attentiveness of theguide’s behavior. When the resolution level is low, usually when the userdislikes the stories or disconfirms the guide’s opinion, the guide providesonly brief facts about a particular location. A high resolution level onthe other hand, will lead the guide to perform more extensive memoryretrieval and story generation, thus, presenting longer stories includingits perspectives on the topic (refer to Lim and Aylett (2007b) for detail).Lastly, selection threshold is the limit competing motives have to crossin order to become active. The higher the selection threshold, the moredifficult it is for another motive to take over and get executed. Thismodulator prevents oscillations between intentions.

Generate and Select Intention

The guide has three possible intentions (goals): UPDATEBELIEF(update its belief about the user’s interests), NEWTOPIC (adjust thestories presentation), and STORYTELLING (perform storytelling). Eachintention becomes active in a different interaction situation. In everyinteraction cycle, two intentions are generated based on needs definedby the built-in motivators state and strength of the intention, which takesaccount of the intention tendencies (likelihood to perform the intention),the guide’s previous experience and expectation of success. The strengthof each intention is a measure of its relevance to the current situation.The intention based on needs is called currentIntention while the intentionbased on strength is called leadingIntention, and they can be identical ordiffer from each other. The pseudocode for generation of these intentionsis presented in Figure 6.

The leadingIntention is determined based on the principle of expectancy-value where the intention with the highest strength is selected. It is obtainedby applying Equations (9)–(16) in sequential order. The calculation inEquations (12)–(16) are applied once to each of the possible intentions.It has to be noted that the choice of parameters in the equations wasfor purely empirical reasons. The equations are the result of mapping theoriginal equations in Dörner (2003) to the context of the Affective Guide



FIGURE 6 Intentions generation and selection.

operations. The presence of a large number of parameters makes the choiceand adjustment of initial parameter values to achieve a plausible startingbehavior tedious. However, once this is achieved, subsequent behaviorsemerge from interaction and learning through experience. In theequations, storytellingTendency, newTopicTendency, and updateBeliefTendencyare the guide’s likelihood to perform storytelling, story adjustment,and update its belief about the user’s interests, respectively. Experiencerepresents the ability of the guide in performing an intention that isaffected by successCount, the number of successes so far in performing thespecific intention divided by totalPerformance, the total number of times theintention was performed. SuccessExpectation is the probability of success foran intention and tendency represents storyTendency, newTopicTendency, orupdateBeliefTendency. Lastly, finalStrength is the strength of the intention afterapplication of decay through multiplication by an intentionFactor:

storytellingTendency = (1 − needForCertainty)

(1 − needForCompetence) (9)

newTopicTendency = needForCompetence2 (10)

updateBeliefTendency = needForCertainty

(1 − needForCompetence) (11)

experience = successCounttotalPerformance

(12)

successExpectation = experience + levelOfCompetence2

(13)



strength = (successExpectation)(tendency) (14)

finalStrength = (strength)(intentionFactor ) (15)

intentionFactor = intentionFactor + 0�05 (16)

The guide selects one of its active intentions for execution dependingon the importance of the needs and the selection threshold value. TheleadingIntention will overtake the currentIntention if and only if its strengthis greater than the strength of the currentIntention plus selection threshold.When either UPDATEBELIEF or NEWTOPIC becomes active, a decay isapplied to the selected intention to stabilize the guide. This is achievedby multiplying a low intentionFactor of 0.25 to the strength. This decreasesthe probability of the same intention being chosen again for a short while,avoiding continuous change of beliefs or adjustment of topics. This factoris increased after the finalStrength is calculated and in subsequent iterationsuntil it reaches a maximum value 1.0.

Run Intention

In order to execute an intention, the guide decides whether to explorefor more information, to design a plan using the available information,or to run an existing plan depending on which intention is selected andits emotional state. Prompt response occurs when there is no story totell or the story for the current location has finished, at which point theguide informs the user of the unavailability of any further story. Planningis performed for STORYTELLING. In the case of UPDATEBELIEF andNEWTOPIC, the guide will explore the database for information so thatappropriate changes to its belief and story topic may take place.

EMOTIONAL MEMORY

Since much of the information we encounter daily holds emotionalsignificance, we view emotional memory as important and necessary forthe Affective Guide. Emotional recollection of past experiences shouldallow the guide to tell more believable and interesting stories. Holdingto this view, the Affective Guide possesses emotional memory including“arousal” and “valence” tags (refer to Lim and Aylett (2007b) for detail),generated through simulation of past experiences. When interactingwith the user, the guide is engaged in meaningful reconstruction of itsown past (Dautenhahn 1998) retrieved from its emotional memory, atthe same time presenting facts about the site of attraction. The user willbe “Walking Through Time” as the guide takes them through the sitepresenting its life experiences and reflecting the emotional impact ofeach experience. Through perspective information and past experiences,



the guide is able to show its attitude towards certain historical events,thus creating awareness in the user about the existence of such aviewpoint. The emotional tag values combine with the guide’s currentbuilt-in motivator values to determine the arousal level and valencelevel (Equations (17) and (18)), allowing the guide to show appropriateexpressions while narrating the stories. For valence calculation, it is againassumed that the guide’s current experience has a greater emotionalimpact than the retrieved experience with a weight ratio of 3:2. The finalvalue falls in the range −0�5 to 0.5.

currentValence = ((1 − (needForCompetence + needForCertainty)/2)

+ experience)/2 (17)

valence = (0�6(currentValence) + 0�4(memoryValence)) − 0�5 (18)

THE STORYTELLING SYSTEM

In the prototype version, the “Los Alamos” site1 of the ManhattanProject has been chosen as the narrative domain because it containedmany characters with different personalities and ideologies that can beused as Affective Guides. The buildings in the “Los Alamos” technical areaare mapped onto the Heriot-Watt Edinburgh campus buildings. Hence, allstories are related to the “Making of the atomic bomb.” It is noteworthythat the Affective Guide includes attitude information only if it is currentlycompetent and highly certain of the user’s interests, in other words, whenits resolution level is high enough. Following are some sample cases ofhow the storytelling system works in concert with the Emergent EmotionModel to perform adaptation of the Affective Guide’s emotions, behavior,and beliefs on the user interests.

Case 1: If the guide’s prediction about the user’s interests is correct(high certainty) and the user perspective is consistent with that ofthe guide (high competence), the guide may experience low-to-mediumarousal level and selection threshold with a medium resolution level. Inthis case, the guide may be diagnosed to experience pride because it canmaster the situation. It is not so easy for another goal to become active.The guide performs an elaboration to the stories and includes its attitude.The guide’s belief about the user’s interests is strengthened, consistentwith Fiedler and Bless’s (2000) argument that an agent experiencingpositive affective states fosters assimilation which supports reliance on andthe elaboration of its existing belief system. It continues to present storiesthat match the user’s interest attributes in its memory.

Case 2: If the guide’s prediction about the user’s interests is right(high certainty), but the user’s perspective is in conflict with the guide’s



perspective (lower competence), then the arousal level of the guide willbe higher than in the previous case. The resolution level decreases whilethe selection threshold increases. The guide has some difficulties in copingwith the differing perspective, but since it has anticipated the situation,it is motivated to concentrate on the specific goal. The increased arousaldue to the user’s disagreement may lead the guide to experience anegative affect (e.g., anxiety) and is taken as a failure feedback, leadingto requirements for new information and inhibiting the application ofcurrent accessible beliefs (Clore and Gasper 2000). Thus, the guideperforms an adjustment to the story topic by modifying its own interest onthe related topic. At the same time, it generates shorter stories that containno or less perspective-based information to prevent further disagreementfrom the user. Furthermore, since it has allocated its resources to gatherinformation and perform story adjustment, fewer resources are availablefor generation of comprehensive stories.

Case 3: In the case that the guide’s prediction about the user’sinterests is wrong (low certainty), but the user’s perspective is consistentwith the guide’s viewpoint (high competence), the arousal level of theguide may be equal to, higher, or lower than the second case dependingon the user’s rating. The selection threshold and the resolution levelremains the same, decreases, or increases accordingly. The guide is stillin control of the situation making the uncertain environment look lessthreatening. Nevertheless, the guide may be disappointed about its wrongprediction. Since the guide is experiencing a negative emotional state,it performs more detailed and substantive processing that will lead tomood repair. Its action involves a high user’s input fidelity where theinformation is utilized to change its beliefs about the user’s interests.Once more, it generates shorter stories that contain only facts becausemost of its resources have been allocated for beliefs update. This is againsupported by Fiedler and Bless’s (2000) discussion that negative statestrigger accommodation processes, allowing beliefs to be updated.

From these examples, it can be observed that UPDATEBELIEF usuallyoccurs when the user dislikes the stories, implying that the guide’sprediction about the user’s interest is incorrect. When this occurs, theguide’s need for certainty increases. Since the guide’s aim is to keepits needs as low as possible, a self-regulatory process takes place. Thisregulative process enables effective action regulation, allowing the guideto perform adaptation by modifying its mental model about the user’sinterests and opinions continuously. NEWTOPIC on the other hand,occurs when the user disagrees with the guide’s argument, that is, whenthe guide’s need for competence is high. In such situations, the guidewill present less perspective-based information about the current topic orchoose a new topic for presentation.



In all cases, the execution will end with story presentation by theguide. The extensiveness of story generation depends on the guide’scurrent emotional state, particularly the resolution level. Normally afterUPDATEBELIEF or NEWTOPIC, a quick response is required as the guidehas a high need for certainty or a high need for competence, in otherwords, it is highly aroused, causing it to perform shallow planning andthus generate short stories that contain only facts or general information.By doing so, the guide not only performs adaptation to user’s interests butalso in terms of its processing strategy based on interaction circumstances.These complex and wide variations of behaviors due to the interactionbetween built-in motivators and modulators are what allow the guide tobe perceived as emotional. Additionally, when stories contain only facts orgeneral information, the possibility of further disagreement from the useris reduced. Meanwhile, the guide uses its updated belief about the user’sinterests for subsequent story generation until another discrepancy occursand this process is continuous.

2D GUIDE CHARACTER

Having internal states, the guide needs a more obvious mechanismfor expression in addition to its behavior modulation. The most commonmeans of expressing emotions is through facial expressions. Adoptingthe “affect as interaction” view (Boehner et al. 2005), we build theAffective Guide based on patterns that are familiar to the user but withoutexplicit labelling of the generated emotional expressions. Ekman andOster (1979) suggested that the individual’s ability to express and judgefacial expressions varies, which further confirms the appropriateness of ourapproach—putting the freedom of interpretation with the observer.

The emergent internal states of the Affective Guide are reflectedthrough a simple 2-dimensional animated talking head. We propose asimple approach of facial expression mapping onto the arousal-valenceemotional space. This approach is flexible and capable of producing awide range of expressions, at the same time making effective use of thescarce resources on the PDA. Research has shown that arousal has thegreatest impact on the eyes (Morris, deBonis, and Dolan 2002; Partala,Jokiniemi, and Surakka 2000). On the other hand, valence affects themouth curvature (Breazeal 2003; Darwin 1948). Additionally, we take intoconsideration the eyebrows, which have been found to be as influential infacial expression recognition (Sadr, Jarudi, and Sinha 2003). We mappedthese three facial features: the eyes, the mouth, and the eyebrows ontothe arousal and valence dimensions, where each dimension influences afacial feature more strongly than the others. The valence value movingfrom negative to positive will move the lip curvature from a downturnU to an upturn U . In the case of extreme pleasure, the cheek raiser is



FIGURE 7 Different facial expressions on the arousal-valence space.

visible below the eyes. In contrast, for extreme displeasure, wrinkles areformed beside the wing of each nostril due to the action of the naso-labialfold. Along the arousal dimension, the size of the eye opening increaseswith increasing arousal and reduces with decreasing arousal. As for theeyebrows, they are influenced by both the arousal and valence values.Under a positive valence, when arousal is low to medium (<0�5), theeyebrows will have a slight V curve. The eyebrows become more andmore relaxed and straightened with increasing arousal. When arousal isvery high (>0�8), the eyebrows will be raised slightly and the raised innereyebrows cause delicate wrinkles to be formed across the forehead. On theother hand, the opposite takes place under negative valence. The resultingfacial expressions along the arousal and valence dimensions are shown inFigure 7.

CONCLUSION

We have proposed an emergent body-mind architecture for emotions,behavior, and belief regulation. Our architecture allows the design of aflexible virtual guide that mediates between internal and external stimuli



to elicit an adaptive behavioral response that serves self-maintenancefunctions. The emotions of the guide are triggered by conditions thatare evaluated as being of significance to its “well-being,” establishingthe desired relation between the guide and its interaction environment.Emotions have not been handcrafted or pregenerated, but emerge frommodulation of cognitive processing, hence, producing a rich set ofexpressions.

The guide presents personalized stories by improvising taking intoconsideration the user’s interests, its own interests, and the previouslytold stories in priority order. The guide expresses its attitude throughperspective information stored in its emotional memories, hence tellingthe user facts as well as its autobiography. Throughout the tour session,the guide performs a continual update of its beliefs about the user’sinterests and adjusts the stories based on the user’s feedback to ensure thatits presentation is always relevant to the user’s expectation. By adaptingits behavior, the guide’s emotional responses mirror those of biologicalsystems, consistent with what a human might expect, hence they shouldseem plausible to a human. The resulting values of cognitive modulationhave a corresponding affective display. The user evaluations (Lim andAylett 2007a; Lim 2007) provide some evidence that the Affective Guidecan indeed make the stories more interesting, engaging, and improveoverall tour experience.

REFERENCES

Almeida, P. and S. Yokoi. 2003. Interactive Character as a Virtual Tour Guide to an OnlineMuseum Exhibition, eds. D. Bearman and J. Trant, North Carolina: Archives & MuseumInformatics. http://www.archimuse.com/mw2003/papers/almeida/almeida.html Last accessedJune 15, 2009.

Bartl, C. and D. Dörner. 1998. Comparing the behavior of PSI with human behavior in the BioLabgame. In: Bamberg: Lehrstuhl Psychologie II, Vol. Memorandum Nr. 32. Bamberg, Germany.

Bertolleti, A. C., M. C. Moraes, and A. C. da Rocha Costa. 2001. Providing personal assistance inthe SAGRES virtual museum. In: Proceedings of Museum and the Web 2001, eds. D. Bearman andJ. Trant, Seattle, WA: Archives & Museum Informatics. http://www.archimuse.com/mw2001/abstracts/prg 95000355.html Last accessed July 31, 2009.

Boehner, K., R. DePaula, P. Dourish, and P. Sengers. 2005. Affect: From information to interaction.In: Critical Computing Conference, eds. O. W. Bertelsen, N. O. Bouvin, P. G. Krogh, and M. Kyng,Aarhus, Denmark: ACM, 59–68.

Breazeal, C. 2003. Emotion and sociable humanoid robots. International Journal of Human-ComputerStudies 59:119–155.

Cañamero, D. 1997. A hormonal model of emotions for behavior control. VUB AI-Lab Memo 97-06.Vrije Universiteit Brussel, Belgium.

Cheverst, K., N. Davies, K. Mitchell, A. Friday, and C. Efstratiou. 2000. Developing a context-aware electronic tourist guide: Some issues and experiences. In: Conference on Human Factorsin Computing Systems, eds. T. Turner and G. Szwillus, Hague, Netherlands: ACM, 17–24.

Clore, G. L. and K. Gasper. 2000. Feeling is believing: Some affective influences on belief.In: Emotions and Beliefs, eds. N. H. Frijda, A. S. R. Manstead, and S. Bem, Cambridge, UK:Cambridge University Press, 10–44.

Damasio, A. 1994. Descartes’ Error: Emotion, Reason and the Human Brain. New York: Gosset/PutnamPress.



Darwin, C. 1948. The Expression of the Emotions in Man and Animals. London: Watts & Co.Dautenhahn, K. 1998. Story-telling in virtual environments. In: Working Notes: Intelligent Virtual

Environments, Workshop at the 13th Biennial European Conference on Aritificial Intelligence, Brighton,UK.

Dörner, D. 2003. The mathematics of emotions. In: Proceedings of the Fifth International Conference onCognitive Modeling, eds. D. Dörner, F. Detje, and H. Schaub, Bamberg, Germany, 75–79.

Dörner, D., J. Gerdes, M. Mayer, and S. Misra. 2006. A simulation of cognitive and emotional effectsof overcrowding. In: Proceedings of the 7th International Conference on Cognitive Modeling, eds. D.Fum, F. Del Missier, and A. Stocco, Trieste, Italy: Edizioni Goliardiche.

Dörner, D. and K. Hille. 1995. Artificial souls: Motivated emotional robots. In: IEEE InternationalConference on Systems, Man and Cybernetics, Vol. 4, 3828–3832.

Ekman, P. and W. V. Friesen. 1982. Measuring facial movement with the facial action coding system.In: Emotion in the Human Face, ed. P. Ekman, 2nd ed., Cambridge: Cambridge University Press,178–211.

Ekman, P. and H. Oster. 1979. Facial expressions of emotion. Annual Review of Psychology 30:527–554.Fiedler, K. and H. Bless. 2000. The formation fo beliefs at the interface of affective and cognitive

processes. In: Emotions and Beliefs, eds. N. H. Frijda, A. S. R. Manstead, and S. Bem, Cambridge,UK: Cambridge University Press, 144–170.

Grammer, K. and E. Oberzaucher. 2006. The reconstruction of facial expressions in embodiedsystems: New approaches to an old problem. ZiF: Mitteilungen 2:14–31.

Gratch, J. and S. Marsella. 2004. Evaluating a computational model of emotion. Journal ofAutonomous Agents and Multiagent Systems (Special issue on the best of AAMAS 2004)11(1):23–43.

Ibanez, J. 2004. An intelligent guide for virtual environments with fuzzy queries and flexiblemanagement of stories. PhD dissertation, Departamento de Ingenieria de la Informacion y lasCommunicaciones, Universidad de Murcia, Murcia, Spain.

Isbister, K. and P. Doyle. 1999. Touring machines: Guide agents for sharing stories about digitalplaces. In: Proceedings of the Narrative Intelligence Symposium, AAAI Fall Symposium Series, eds. M.Mateas and P. Sengers, Massachusetts: AAAI Press, 93–100.

Lim, M. Y. 2007. Emotions, behaviour and belief regulation in an intelligent guide with attitude.PhD dissertation, School of Mathematical and Computer Sciences, Heriot-Watt University,Edinburgh, UK.

Lim, M. Y. and R. Aylett. 2007a. Feel the difference: A guide with attitude! In: Proceedings of the7th International Working Conference on Intelligent Virtual Agents, eds. C. Pelachaud, J. Martin, E.André, G. Chollet, K. Karpouzis, and D. Péle, Paris, 317–330.

Lim, M. Y. and R. Aylett. 2007b. Narrative construction in a mobile guide. In: Proceedings of the4th International Conference on Virtual Storytelling, eds. M. Cavazza and S. Donikian, Saint Malo,France, 51–62.

Lim, M. Y. and R. Aylett. 2007c. A new approach to emotion generation and expression. In:Proceedings of the Doctoral Consortium. The 2nd International Conference on Affective Computing andIntelligent Interaction, eds. R. Cowie and F. de Rosis, Lisbon, Portugal, 147–154.

Lim, M. Y., R. Aylett, and C. M. Jones. 2005. Emergent affective and personality model. In:Proceedings of the 5th International Working Conference on Intelligent Virtual Agents, eds. T.Panayiotopoulous, J. Gratch, R. Aylett, D. Ballin, P. Olivier, and T. Rist, Kos, Greece, 371–380.

Morris, J. S., M. deBonis, and R. J. Dolan. 2002. Human Amygdala responses to fearful eyes.NeuroImage 17:214–222.

Nass, C., J. Steuer, and E. R. Tauber. 1994. Computers are social actors. In: Proceedings of the SIGCHIConference on Human Factors in Computing Systems: Celebrating Interdependence, eds. B. Adelson, S.Dumais, and J. Olson, Boston, MA: ACM, 72–78.

O’Grady, M. J., R. P. O’Rafferty, and G. M. P. O’Hare. 1999. A tourist-centric mechanism forinteracting with the environment. In: Proceedings of the First International Workshop on ManagingInteractions in Smart Environments. Dublin, Ireland: Springer, 56–67.

Ortony, A., G. Clore, and A. Collins. 1988. The Cognitive Structure of Emotions. Cambridge, UK:Cambridge University Press.

Partala, T., M. Jokiniemi, and V. Surakka. 2000. Pupillary responses to emotionally provocativestimuli. Eye Tracking Research and Applications Symposium, ed. A. T. Duchowski, Palm BeachGardens, Florida: ACM, 123–129.



Petrelli, D. and E. Not. 2005. User-centred design of flexible hypermedia for a mobileguide: Reflections on the hyperaudio experience. User Modeling and User-Adapted Interaction15(3–4):303–338.

Pezzulo, G. and G. Calvi. 2007. Rational agents that blush. In: Proceedings of the 2nd InternationalConference of Affective Computing and Intelligent Interaction, eds. A. Paiva, R. Prada, and R. Picard,Lecture Notes in Computer Science. Springer, 374–385.

Picard, R. W. 1997. Affective Computing. Boston: MIT Press.Russell, J. A. 1997. Reading emotions from and into faces: Resurrecting a dimensional-contextual

perspective. In: The Psychology of Facial Expression, eds. J. A. Russell and J. M. Fernaández-Dolz,Cambridge: Cambridge University Press, 295–320.

Sadr, J., I. Jarudi, and P. Sinha. 2003. The role of eyebrows in face recognition. Perception32:285–293.

Stock, O. and M. Zancanaro. 2007. PEACH – Intelligent Interfaces for Museum Visits. Berlin: Springer.Sumi, Y., T. Etani, S. Fels, N. Simone, K. Kobayashi, and K. Mase. 1998. C-MAP: Building a context-

aware mobile assistant for exhibition tours. In: Community Computing and Support Systems, SocialInteraction in Networked Communities, ed. T. Ishida, London, UK: Springer-Verlag, 137–154.

Turrini, P., J. C. Meyer, and C. Castelfranchi. 2007. Rational agents that blush. In: Proceedings ofthe 2nd International Conference of Affective Computing and Intelligent Interaction, eds. A. Paiva,R. Prada, and R. Picard, Lecture Notes in Computer Science. Springer, 314–325.

Velásquez, J. 1998. A computational framework for emotion-based control. In: Proceedings of theGrounding Emotions in Adaptive Systems Workshop, SAB ’98, eds. D. Canamero, C. Numaoka, andP. Petta, Zurich, Switzerland, 62–67.

NOTE

1� http://www.lanl.gov/


PLEASE SCROLL DOWN FOR ARTICLE - h. Wmyl/Papers/AAIPublished.pdf · PLEASE SCROLL DOWN FOR ARTICLE...

Documents

Transcript of PLEASE SCROLL DOWN FOR ARTICLE - h. Wmyl/Papers/AAIPublished.pdf · PLEASE SCROLL DOWN FOR ARTICLE...