1

Affect and Personality inInteraction with Ubiquitous

SystemsProfessor Ruth Aylett

Vision Interactive Systems & GraphicalEnvironments

MACS, Heriot-Watt Universitywww.macs.hw.ac.uk/~ruth

Summary of programme

Introduction and overview (today) Affective outputs

– speech, language, gesture, facial expressions, music, colour

Affective/Personality models and action-selectionapproaches

Affective inputs Applications

– Embodied Conversational Characters, Intelligent Virtual,Agents, human-robot interaction

Evaluation approaches

Today’s topics

Describing emotion Music Colour Shape and form

Thanks to Catherine Pelachaud!

Displaying emotion

Emotions can be shown via– Acoustic and visual behaviors: facial expression, voice,

gesture, posture– Behavior expressivity: voice and body movement quality– Music– Colour

Reasons to display emotional state:– Create affective awareness– Create social relationship– Engage user in communication

But how do we know what to output?– Some systematic description of emotion?

2

Affect dispositions: nervous, anxious, reckless, morose, hostile

Preferences/Attitudes: liking, loving, hating, valuing, desiring

Interpersonal stances: distant, cold, warm, supportive, contemptuous

Moods: cheerful, gloomy, irritable, listless, depressed, buoyant

Emotions: angry, sad, joyful, fearful, ashamed, proud, elated, desperate

Beh

avio

r im

pact

Rap

idity

o

f ch

an

ge

Ap

pra

isal

elic

itatio

n

Even

t fo

cu

s

Syn

ch

ro-

niz

atio

n

Du

ratio

n

In

ten

sity

Design Features

Types of Affect

Affect dispositions: nervous, anxious, reckless, morose, hostile

Preferences/Attitudes: liking, loving, hating, valuing, desiring

Interpersonal stances: distant, cold, warm, supportive, contemptuous

Moods: cheerful, gloomy, irritable, listless, depressed, buoyant

Emotions: angry, sad, joyful, fearful, ashamed, proud, elated, desperate

Beh

avio

r im

pact

Rap

idity

o

f ch

an

ge

Ap

pra

isal

elic

itatio

n

Even

t fo

cu

s

Syn

ch

ro-

niz

atio

n

Du

ratio

n

In

ten

sity

Design Features

Types of Affect

Defining types of affective statesScherer et al.,Univ. Geneva

Russell’s system

CircumplexModel of Affect– Russell 1980

two components(1) pleasure-

displeasureVALANCE

(2) arousal-sleepAROUSAL

adventurous

triumphant

lusting

ambitious conceited

bellicose

self-confident

courageous feelingsuperior

convinced

light-hearted

enthusiastic

determined amused

passionate

expectant

elated

interested

joyous

excited

hostile

envious hateful

enraged defiant

contemptuous jealous

angry

disgusted

loathing indignant

impatient suspicious

bored

distrustful

startled

insulted

bitterdiscontented

feel well impressed disappointed

EXCITED •

amourous astonished apathetic dissatisfied

confident takenaback content hopeful

relaxed longing

solemn attentive

worried

uncomfortabledespondent

feel guilt

languid ashamed desperate

friendlycontemplative

pensiveembarrassed

polite serious

conscientious

peaceful

reverentempathic

melancholic

hesitant wavering

anxious

lonely

doubtful

sad dejected insecure

DEPRESSED •

•ASTONISHED

•AROUSED

•DELIGHTED

HAPPY •

PLEASED•GLAD •

SERENE •

CONTENT• AT EASE•SATISFIED• RELAXED

• CALM •

SLEEPY •

• TENSE•ALARMED• ANGRY • AFRAID

ANNOYED•

•DISTRESSED

FRUSTRATED•

MISERABLE •

• SAD

• GLOOMY

• TIRED

• BORED

DROOPY •

Sherer’s descriptive framework

Scherer et al.Univ. Geneva

Happy

Angry

Sad

Hi Power/Control

Conducive

Obstructive

Lo Power/Control

Active

Positive Negative

Passive

An empirical subset suitable for describingemotions in human-machine interaction

Preliminary list of 55 terms, fromHUMAINE summer school 2004, Belfast:

stress, annoyance, boredom, panic, impatience, disapproval, hotanger, anxiety, disappointment, fear, satisfaction, sadness,surprise, shock, amusement, worry, excitement, pleasure, coldanger, interest, effervescent happiness, nervousness, approval,embarrassment, distraction, disagreeableness, disgust, despair,indifference, neutrality, hurt, friendliness, weariness, relief,confidence, contentment, shame, contempt, affection, sympathy,relaxation, mockery, pride, resentment, calm, guilt, jealousy,determination, serenity, coldness, cruelty, hopeful, wariness,greed, admiration

3

Affective MusicBresin, KTH Sweden

Simulation of emotions in musicperformance– Mapping between expressive acoustic cues and

emotions

Visualization of musical expression– Colours– Facial expressions

• TENDERNESSslow mean tempo (Ga96)slow tone attacks (Ga96)low sound level (Ga96)small sound level variability(Ga96)legato articulation (Ga96)soft timbre (Ga96)large timing variations (Ga96)accents on stable notes (Li99)soft duration contrasts (Ga96)final ritardando (Ga96)

• HAPPINESSfast mean tempo (Ga95)small tempo variability (Ju99)staccato articulation (Ju99)large articulation variability (Ju99)high sound level (Ju00)little sound level variability (Ju99)bright timbre (Ga96)fast tone attacks (Ko76)small timing variations (Ju/La00)sharp duration contrasts (Ga96)rising micro-intonation (Ra96)

• ANGERhigh sound level (Ju00)sharp timbre (Ju00)spectral noise (Ga96)fast mean tempo (Ju97a)small tempo variability (Ju99)staccato articulation (Ju99)abrupt tone attacks (Ko76)sharp duration contrasts (Ga96)accents on unstable notes (Li99)large vibrato extent (Oh96b)no ritardando (Ga96)

• SADNESSslow mean tempo (Ga95)legato articulation (Ju97a)small articulation variability (Ju99)low sound level (Ju00)dull timbre (Ju00)large timing variations (Ga96)soft duration contrasts (Ga96)slow tone attacks (Ko76)flat micro-intonation (Ba97)slow vibrato (Ko00)final ritardando (Ga96)

• FEARstaccato articulation (Ju97a)very low sound level (Ju00)large sound level variability(Ju99)fast mean tempo (Ju99)large tempo variability (Ju99)large timing variations (Ga96)soft spectrum (Ju00)sharp micro-intonation (Oh96b)fast, shallow, irregular vibrato(Ko00)

Positive Valence

Negative Valence

High ActivityLow Activity

From Juslin (2001)

R. Bresin

Expressive cues

Tempo

Loudness

Timbre

Encoding Decoding

Cue Utilization Cue Utilization

The Listener

Articula.

The Performance The Performer intention expressive cues judgment

Accuracy

others

Anger Anger

rPerformer rListener

Matching

.87

.26 .47 .63

-.26

.22 .55 .61

-.39

.92

Lens model: quantifies the expressivecommunication between performer and listener

R. Bresin

Example: SADNESS

Expressive Cue Analysis Synthesis (Director Musices)

Tempo Slow Tone IOI is lengthened by 30%

Sound level Moderate or low Sound level is decreased by 6 dB

Articulation Legato Tone duration = Tone IOI

Time deviations Moderate Duration Contrast Rule (k = -2)

Phrase Arch Rule applied on phrase level (k = 1.5)

Phrase Arch Rule applied on sub -phrase level (k = 1.5)

Final ritardando Yes Obtained from the Phrase Arch Rule

R. Bresin

4

Better Polyphonic RingtonesMOODIESBresin, KTH

OriginalClassyHappyRomanticAggressive

OriginalClassyHappyRomanticAggressive

OriginalClassyHappyRomanticAggressive

ColdplayTalk

La Linea Hayfa

R. Bresinwww.notesenses.com

Colour, Movement, Shape

Color and EmotionBresin et al, KTH

• Perceptual study: Link musical performances to colours• Performances with different emotional intentions• Set of colour nuances in hue, brightness, saturation• Result of perceptual study:

From Bresin

BRIGHTNESSObserved tendency:Minor tonality Low brightness (Darkcolours)Major tonality High brightness (Lightcolours)

HUEHappiness YellowFear BlueSadness Violet & BlueAnger RedLove Blue & Violet

Visualization of MusicalExpression

Tool for real-time visual feedback to expressiveperformance

Mapping between acoustic cues and emotions ExpressiveBall: Mapping of emotions and colors GretaMusic: Mapping of emotions and facial

expressions– music emotion facial expression– music volume spatial and power– music tempo temporal and overall activation– music articulation fluidity

5

The ExpressiBallBresin, Juslin, KTH

X Tempo Color EmotionY Sound level Shape ArticulationZ Attack velocity & Spectrum energy

Lo

ud

So

ft

Slow Fast

StaccatoAngryFast attackHigh energy

Lo

ud

So

ft

Slow Fast

LegatoSadSlow attackLow energy

From Bresin

GretaMusic Bresin, KTH – Mancini, Pelachaud U Paris8

Mutual Interaction

• Interactive virtual dancer:• dance together with the user to the beat of the music• adapt its performance to whatever the human user is doing

- beat detection toalign dance withmusic tempo- agent’s movementchosen from databaseof capture movements

Affective DiaryHöök et al, SICS

From Höök

•Diary: express inner thoughts and record experiences of pastevents•Affective diary:

• capture emotional experience over time via mobile phone• replay of the experience• reflect on the experience

6

Affective DiaryHöök et al, SICS

”[pointing at thefirst slightly redcharacter] And thenI become like this,here I am kind of, Iam kind of bothhappy and sad insome way andsomething like that.I like him and thenit is so sad that wesee each other solittle. And then Icannot really showit.”

From Höök

eMoto - Expressing emotions ina digital world

Sundström, Ståhl, Höök, SICS-KTH

eMoto: mobile messaging service for sendingand receiving affective messages

Use affective gestures of users to convey theemotional content of their messages

eMoto - Expressing emotions ina digital world

Sundström, Ståhl, Höök, SICS-KTH

Input: movement detection through pen Output: colours, shapes and animations on

mobile

eMoto – ExampleSundström, Ståhl, Höök, SICS-KTH

Bored Excited Happiness

7

I-ShadowsPaiva et al, Gaips

Help children tobuild the virtual andreal world ofInteractive Shadows

Create a learningenvironment wherechildren will be ableto build logicalnarratives on-the-fly

AINI (Anticipatory Believability)Martinho, Paiva Gaips

Agent with autonomous anticipatory mechanism Study of the relation between anticipation and

emotion Prediction of the next sensor value of agent

– interpretation of the mismatch between sensation andanticipation to direct both the focus of attention and theexpression of emotions.

QUESTIONS?