Issues in Emotion-oriented ComputingTowards a shared understanding

Marc Schröder & Roddy Cowie

http://emotion-research.net

humaine - emotion-research.net 2

HUman-MAchine Interaction Network on Emotions

EU network of excellence 2004-2007scientific and technological groundwork for emotion-oriented computing

interdisciplinary understandingprincipled approaches

humaine - emotion-research.net 3

Sub-areas in Emotion-oriented Computing

Empirical evidenceEmotion

detection

planning

Emotional language

evaluation

Description of

emotion

Ethicsof emotion

Emotion detection

Planning action

Theory ofemotional processes

Raw empirical

data

Emotion incomplexmedia

Usability / evaluation

Synthesis of basicsigns

Application oriented

Core technologies

Psychologically oriented

humaine - emotion-research.net 4

Sub-areas in Emotion-oriented Computing

Empirical evidenceEmotion

detection

planning

Emotional language

evaluation

Description of

emotion

Ethicsof emotion

Emotion detection

Planning action

Theory ofemotional processes

Raw empirical

data

Emotion incomplexmedia

Usability / evaluation

Synthesis of basicsigns

Application oriented

Core technologies

Psychologically oriented

Theory ofemotional processes

humaine - emotion-research.net 5

Theory of Emotional Processes

what to study: states that are relevant to emotion-oriented computingterminological clarification: clear definition of “emotions”, “moods”, …practically useful methods for describing emotional states

humaine - emotion-research.net 6

“Less than 10% of human life is truly unemotional”

Cowie et al., QUB:reality TV show “Cast away”ratings of unedited tapes

truly unemotional episodesare rareemotion in the strong senseis rare

need concepts to describe the ~80% “elements of emotion”

humaine - emotion-research.net 7

Aspects of “emotional life”

Cowie, O’Neill et al., QUB

emergent emotion (full-blown)emergent emotion (suppressed)moodpartial emotion (topic shifting)partial emotion (simmering)attitudeestablished emotioninterpersonal stancesaltered states of arousal/control/seriousness

humaine - emotion-research.net 8

Aspects of “emotional life”

Cowie, O’Neill et al., QUB

emergent emotion (full-blown)emergent emotion (suppressed)moodpartial emotion (topic shifting)partial emotion (simmering)attitudeestablished emotioninterpersonal stancesaltered states of arousal/control/seriousness

DefinitionThis refers to states between a mood and an emergent emotion, where a person is in a single kind of emotional state, but it is not fixed on a single thing or issue – instead the emotion attaches to one object, then another, then another, in quite rapid succession. Example“Angry and tired from previous day. Getting annoyed when styling hair because it wouldn’t stay. Angry at dog trying to get out, and things at work.”

humaine - emotion-research.net 9

Labels for “emotional life”

Negative & forceful Anger Annoyance Contempt Disgust Irritation Negative & not in control Anxiety Embarrassment Fear Helplessness

Powerlessness WorryNegative thoughts Doubt Envy Frustration Guilt ShameNegative & passive Boredom Despair Disappointment Hurt Sadness

Agitation Shock Stress TensionPositive & lively Amusement Delight Elation Excitement Happiness

Joy Pleasure

Caring Affection Empathy Friendliness Love Positive thoughts Courage Hope Pride Satisfaction TrustQuiet positive Calm Content Relaxed Relieved Serene

Reactive Interest Politeness Surprise

Cox et al.: List consolidated from Geneva, LIMSI, Belfast sources

humaine - emotion-research.net 10

Empirical evidenceEmotion

detection

planning

Emotional language

evaluation

Description of

emotion

Ethicsof emotion

Emotion detection

Planning action

Theory ofemotional processes

Raw empirical

data

Emotion incomplexmedia

Usability / evaluation

Synthesis of basicsigns

Application oriented

Core technologies

Psychologically oriented

Sub-areas in Emotion-oriented Computing

Raw empirical

data

humaine - emotion-research.net 11

Emotions in Empirical Data

Acted

Induced

TV interviews

In action

humaine - emotion-research.net 12

Emotions in Empirical Data

“supportive” datato fill in gaps in theory

“provocative” datato challenge preconceptions

develop suitable labelling schemes

humaine - emotion-research.net 13

Empirical evidenceEmotion

detection

planning

Emotional language

evaluation

Description of

emotion

Ethicsof emotion

Emotion detection

Planning action

Theory ofemotional processes

Raw empirical

data

Emotion incomplexmedia

Usability / evaluation

Synthesis of basicsigns

Application oriented

Core technologies

Psychologically oriented

Sub-areas in Emotion-oriented Computing

Emotion detection

humaine - emotion-research.net 14

Emotion Detection: Multimodal Features

humaine - emotion-research.net 15

Emotion Detection: Selected Challenges

Integration of modalitiesdifferent time scales for analysis

face = still picturesvoice = discrete chunks on time scalephysiology = slow, continuous variation over time

Measure of success?same errors as humans?

dialogue system

more accurate than humans?stress detection

humaine - emotion-research.net 16

Empirical evidenceEmotion

detection

planning

Emotional language

evaluation

Description of

emotion

Ethicsof emotion

Emotion detection

Planning action

Theory ofemotional processes

Raw empirical

data

Emotion incomplexmedia

Usability / evaluation

Synthesis of basicsigns

Application oriented

Core technologies

Psychologically oriented

Sub-areas in Emotion-oriented Computing

Synthesis of basic signs

humaine - emotion-research.net 17

Greta agentPelachaud et al.,Univ. Paris 8

Emofilt speech synthesis filterBurkhardt, T-Systems

Synthesis of Signs of Emotion:Parametrisable Models of Body, Face and Voice

humaine - emotion-research.net 18

Synthesis of Signs of Emotion

make embodied conversational agents (ECAs) behave emotional

required key capabilities:Coordination of signs in multiple modalities ExpressivityCognitive influences on action (attention model)Creating affective awarenessBackchannel & Feedback behaviour

specify architecture requirementstowards proof-of-concept sub-systems

humaine - emotion-research.net 19

Attention Modelling for ECAs

crucial for initiating and maintaining communicationvisual salience

contrast, color, …facesemotionalexpressions

Peters et al., Univ. Paris 8

humaine - emotion-research.net 20

Empirical evidenceEmotion

detection

planning

Emotional language

evaluation

Description of

emotion

Ethicsof emotion

Emotion detection

Planning action

Theory ofemotional processes

Raw empirical

data

Emotion incomplexmedia

Usability / evaluation

Synthesis of basicsigns

Application oriented

Core technologies

Psychologically oriented

Sub-areas in Emotion-oriented Computing

Planning action

humaine - emotion-research.net 21

Emotion in Planning and Action

conceptual and practical work on cognitive architectures with emotion

low-level / sub-symbolic / reactivehigh-level / symbolic / deliberativehybrid: bridging the gap

humaine - emotion-research.net 22

Low-level Perception-action Loop:Emergent “aggression”

robots need two resources to survivewander around avoiding objectsblue robot: bumper sensitivity reduced when deficit too largeresulting behaviour: pushing the otheraway from resourceethological analysis: “aggressive” behaviour

Cañamero et al.,Univ. Hertfordshire

humaine - emotion-research.net 23

Low-level Perception-action Loop:Imprinting a “desired perception”

Imprint visual perception of “optimal” box distanceRobot will follow box in this distance,goal = re-establish desired perceptionK. Lorenz

Blanchard & Cañamero, Univ. Hertfordshire

humaine - emotion-research.net 24

High-level: Emotions in BDI models

Reasoning about emotionsinterpret emotional impactof previous dialogue moveguess possible emotional impactof next dialogue move

Based on an appraisal model (e.g., OCC)link events to emotional meaning

humaine - emotion-research.net 25

Empirical evidenceEmotion

detection

planning

Emotional language

evaluation

Description of

emotion

Ethicsof emotion

Emotion detection

Planning action

Theory ofemotional processes

Raw empirical

data

Emotion incomplexmedia

Usability / evaluation

Synthesis of basicsigns

Application oriented

Core technologies

Psychologically oriented

Sub-areas in Emotion-oriented Computing

Emotion in

complex media

humaine - emotion-research.net 26

Emotion in Complex Media

closest to applicationsexploratory work

automatic generation of potentially funny slogansplaying with semantic ambiguities in languagehuman has to decide which slogans are funnyExample: slogan for soft drinks: “Thirst come, thirst served!” (Stock et al., itc-IRST)

persuasive ECA communicationpolitenesslying behaviour

humaine - emotion-research.net 27

Generating Lying Expressions in an ECA

GAMBLE dice gameRehm et al.,Univ. Augsburg

humaine - emotion-research.net 28

Empirical evidenceEmotion

detection

planning

Emotional language

evaluation

Description of

emotion

Ethicsof emotion

Emotion detection

Planning action

Theory ofemotional processes

Raw empirical

data

Emotion incomplexmedia

Usability / evaluation

Synthesis of basicsigns

Application oriented

Core technologies

Psychologically oriented

Sub-areas in Emotion-oriented Computing

Usability/ evaluation

humaine - emotion-research.net 29

Usability of Emotion-oriented Systems

existing usability criteria do not applye.g., control, predictability, transparency

Develop criteria and methods to evaluate affective interaction systems

“sensual” feedback methodsWizard-of-Oz studies extended think-aloud protocol

humaine - emotion-research.net 30

“Sensual” Feedback Methods for Usability Testing

SenToy puppetPaiva et al., INESC-ID

Colour and shapeHöök et al., KTH

humaine - emotion-research.net 31

Wizard of Oz Method: ECA-based Dialogue

study user behaviouruser-centered design loop

design->test->redesign

explore affective interactionsocial relationship / empathy

de Rosis et al.,DI-BARI

User interface

Wizard interface

humaine - emotion-research.net 32

Empirical evidenceEmotion

detection

planning

Emotional language

evaluation

Description of

emotion

Ethicsof emotion

Emotion detection

Planning action

Theory ofemotional processes

Raw empirical

data

Emotion incomplexmedia

Usability / evaluation

Synthesis of basicsigns

Application oriented

Core technologies

Psychologically oriented

Sub-areas in Emotion-oriented Computing

Ethics of emotion

humaine - emotion-research.net 33

Ethics in HUMAINE

Data collection: emotional data is sensitivePotential abuse of emotionally competent systems

automatic persuaderssurveillance systems…

Address ethical issues proactively

humaine - emotion-research.net 34

Ethics in HUMAINE

Ethics research is new to this communityunrealistic to formulate a closed set of criteriaapproach: “Principlism” (Beauchamps & Childress, 2001)

Four Level Modelnonmaleficence, autonomy, beneficance, justiceuniversally shared ethical standardsrelies on “seeing things right”

Install ethical committeebecause every case is different

humaine - emotion-research.net 35

Conclusions

Working on these different levelsremoving obstacles in cooperation(“make things fit together”)building basis for future worksuggesting sensible research agendas

Applications Coretechnologies

Psychologicaltheories ofemotion

humaine - emotion-research.net 36