1

What is an emotion?

• The recognition of a change of feeling

• God’s punishment for disobedience -- St Augistine

• A brain process that computes the value of an experience --- Le Doux

• A word we assign to certain configuration of bodily states, thoughts, and

situational factors – Feldman Barrett.

2

What is emotion NOT?

From Scherer

3

Do emotion’s “exist”?

• Are emotions “natural kinds”

– Are they reflected in the structure of nature or are they concepts

we impose on a lose collection of related structures

• Basic/Discrete emotion view (e.g. Ekman, Le Doux…)

– Some of what we call emotions correspond to brain circuits • Joy, Hope, Rfear, (Joy, Hope, Fear, Sadness, Anger, Disgust)

– Distinguished from more complex affective states

• Love, optimism, awe, remorse, shame

E.g. Le Doux fear

circuit

4

Do emotion’s “exist”?

• Core Affect View (Russell, Feldman Barrett)

– Emotions don’t exist as discrete circuits

– Emotions correspond differences in body state distinguished in

terms of broad dimensions (e.g., valence and arousal)

5

What is an emotion?

• Component view

– episode of massive synchronized recruitment of mental and

somatic resources allowing to adapt to cope with stimulus event

subjective appraised as being highly pertinent to needs, goals,

and values of the individual. -- Klaus Scherer

• Components

– Cognitive elements: Goals, needs, values / Subjective appraisal

– Somatic elements: Physiological activation / Display / Action

tendency

– Feelings: some authors distinguish these cognition

• Prototypical characteristics

– High intensity

– Synchronization across components

– Adaptive / Coping

Appraisal Theory

Affective Computing: Lecture 2

Stacy Marsella

Jon Gratch

7

Emotion

CognitiveEmotion is a mental phenomena

• Arises from physiological & cognitive processes

• Associated with key cognitive functions– Focusing mental, sensory resources

– Influencing beliefs

– Informing decision-making

– Preparing action and reaction

– Learning and long-term adaptation

8

Emotion

Emotion is a physical phenomenon

• Emotion processes associated with specific

physical behaviors

– Facial expression

– Body language and posture

– Voice

– Behavioral dynamics

Cognitive

Physical

9

Emotion

Emotion is a social phenomenon

Emotion is a signal that influences the

behavior of others

– Emotional contagion

– Social referencingPhysical

Social

Cognitive

10

Why our we interested in modeling emotion?

Goal is to develop unified computational framework for modeling, simulating, explaining and exploiting these phenomena

• To inform intelligent system design

– By drawing on insights from emotion’s cognitive/social function

– Simon, 1968; Minsky; Scheutz, &Sloman, 2001

• To inform emotion theory

– By concretizing theory and developing methodological tools

• To drive applications

– Education, Training and Health Interventions

Cognitive

Physical

Social

Unifie

d U

nd

ers

tan

din

g

11

– Application Areas:

– Social Learning Environments

– Gratch&Marsella04; Diaz04 ….

– User modeling

– Infer student motivational state (Conati&MacLaren , 2004)

– Entertainment

– “Creating Emotion in Games”

Social Learning Environments

Explore emotionally charged social

interactions in safety of VR

Virtual Role Play

Learner interacts with Virtual

Humans

Virtual Humans

• Interact with users– Computer-generated simulations of humans

– Play role of mentors, peers, competitors, etc.

• Communicate thru speech and gesture– Coordinated gestures and non-verbal

communication

• Reason about environment– Behavior not pre-scripted

– Reason about possible responses to events

• Understand & express emotions– Affects way they perform tasks & interact socially

– Affects gaze, face, gestures, posture

ICT Movie

Health Interventions •Mental health (Marsellaet al)

• Autism (Tartaro&Cassell)

Diagnosis & Assessment•ADHD Assessment (Rizzo et. al)

Education and Training•Intelligent tutoring systems (Lester et al; Graesser)• Health care provider training (Lok et al)

Virtual Human Applications

Entertainment• Façade (Mateas and Stern)

Informal Science Education•Museum Guides (Boston Science Museum, Cooper-Hewett)

Theoretical Issues

Theoretical Views

Categorical vs dimensional – Categorical

Tomkin’s 8: fear, anger, anguish, joy, disgust, surpass, interest and shame

Ekman’s six basic emotions

– Culturally universal

– Dimensional: Russell’s circumplex: Arousal & Valence (Pleasure)

Russell &Mehrabian’s PAD model: Pleasure, Arousal & Dominance

– Appraisal Theories

Utility for computational work?– Virtual Humans, Behavior animation, Speech

Theoretical Views (cont)

Function of Emotion– Stress on Social Coordination

E.g., Parkinson

– Stress on Individual Regulation and Adaptation

E.g., Lazarus

Function of Emotional Expression– Social Signal

– Reflection of internal state

– Part of adaptive behavior

Example: Eyelid and eyebrow lift

Theoretical Issues (cont):

Top down– Emotion arises from cognitive processes

– Appraisal Theories Stimulus > Cognition/Appraisal > feeling

Bottom up– Physiological and perceptual processes drive

– Zajonc Stimulus > unconscious affect > feeling

– James Stimulus > response > feedback > feeling

Pathways go both ways– May differ for different emotions

20

Dimensional

Appraisal

Models

ARElliott EM

Neal Reilly

FLAMEEl Nasr

EmileGratch

CBIMarsella

Lazarus

EMAGratch/Marsella

FearNot!Dias

NAFOMarinier

CathexisVelásquez

Frijda

Armony

Scheultz

Mehrabian ALMAGebhard

MAXBecker

MAMIDHudlicka

ACRESSwagerman

OCC

WillMoffat

Sloman

Neuro/Ethology

(e.g., LeDoux,

Damasio)

Cognitive

Architecture

PerspectiveHOTCO

Thagard

NML1Beaudoin

Minder1Wright

Alvila-Garcia,

Canamero

21

Diagram comments

• Draw on Different Theories– Grounded in different experimental techniques

– Appraisal Theories dominant

• Creating models from particular theories versus pulling from different theories

• Models of specific affective phenomena versus more general models

• Built for different Applications

• Different computational techniques

• Lineage

Modeling the cognitive antecedents and

consequences of emotion

Cognitive

• Top down theories– Cognition influences emotion

– Appraisal Theory (Arnold, Lazarus, Frijda, Scherer, Ortony et al.)

Emotion arises from an evolving subjective interpretation of person’s relation to their environment and informs cognitive and physical acts

Magda Arnold

Theories of cognitive emotion

(act on world) (act on beliefs)

Appraisal Variables

CopingStrategy

ActionTendencies

“Affect”Physiological

Response

Appraisal

Coping

Smith and Lazarus91 cognitive-motivational-emotive system

Problem-Focused Emotion-Focused

EnvironmentGoals/Beliefs/

Intentions

Desirability

Expectedness

Controllability

Causal Attribution

Appraisal Theory

(act on world) (act on beliefs)

Appraisal VariablesAction

Tendencies“Affect”

PhysiologicalResponse

Appraisal

Problem-Focused Emotion-Focused

EnvironmentGoals/Beliefs/

Intentions

Desirability

Expectedness

Controllability

Causal Attribution

Take action

Seek support

Form/drop goal

Form/drop belief

Form/drop intention

Coping

Smith and Lazarus91 cognitive-motivational-emotive system

Appraisal Theory

Appraisal

Appraisal

• Appraisal = Situation assessment

– Compare beliefs, desires and intentions

with

external circumstances

Internal

goals, beliefs

External

World Appraisal

Appraisal

• Characterize via appraisal variables– Desirability

• Does this event help/hurt my goals

– Likelihood

• How likely is it that this event will occur

– Unexpectedness

• Was this event expected

– Causal attribution (causality, agency, blame/credit)

• Who deserves blame for causing the event

– Coping potential (controllability, adaptability)

• What chance do I have for dealing with this event

– Urgency

• Superset of criteria considered by intelligent systems

– Decision theory: desirability, likelihood

– Scheduling: desirability, urgency

Appraisal

• Emotions defined in terms of

configurations of appraisal variables

– Undesirable, Uncertain Fear

– Desirable, Certain Joy

– Undesirable, Caused-by(Other) Anger-at(Other)

Problem-

focusedEmotion-

focused

• Coping = Response strategy

– Characterized by ontology of coping

strategies

Emotion

Coping Strategies

Appraisal

AppraisalCoping

Internal

goals, beliefs

External

World

• Problem-focused (act on the world)

– Action execution

– Planning

– Seek instrumental social support

• Analogous to:– Deliberative or reactive problem solving

– Team negotiation

Coping Strategies

• Emotion-focused (act on belief)– Denial

– Find positive meaning

– Resignation

– Shift blame

– Distancing

• Not typically considered by intelligent systems– More than a decision (e.g. abandon current plan)

• Provides self-justification for why

• Related to motivational / explanatory coherence

• Leads to persistent change in behavior

• Helps find ecological niche

Coping Strategies

Appraisal Theory as

architectural specification

• Provides high-level requirements

• How do we map this into an architecture – How do we represent the person-environment relation?

– How do appraisal processes operate over this representation?

– What is the relation between the processes of

• Appraisal

• Cognition

• Coping

– How do these interactions interact/unfold over time?

EMAGratch & Marsella

Planning Perception Dialogue Action

Causal Interpretation:

Working memory of plans, beliefs, desires, intentions

Cognitive Operations (inference)

Past Events Future PlansPresent

EMA Model of Appraisal and CopingCognition as multiagent planning/envisionment

Future ActCause: self

Intend: yes

Probability: 50%

GoalUtility: 50

Probability: 50%Intend-that: True

Goal

Utility: 50 Probability: 100%

Belief: FalsePast ActCause: Other

Intend: yes

Prob: 100%Facilitates

Inhibits

Future ActCause: self

Intend: yes

Probability: 50%

GoalUtility: 50

Probability: 50%Intend-that: True

Past Present Future

Goal

Utility: 50 Probability: 100%

Belief: False

Past ActCause: Other

Intend: yes

Prob: 100%

ChallengeDesirability: 50

Likelihood: 50%

Causal Attribution: self

Coping Potential: Moderate

Emotion: Hope(25)

ThreatDesirability: -50

Likelihood: 100%

Causal Attribution: Other

Coping Potential: moderate

Emotion: Anger(50)

FacilitatesInhibits

Appraisal

ThreatDesirability: -100

Likelihood: 50%

Causal Attribution: Other

Coping Potential: Low

Emotion: Sadness(50)

Past Present Future

Goal

Utility: 50 Probability: 100%

Belief: False

Past ActCause: Other

Intend: yes

Prob: 100%

ThreatDesirability: -50

Likelihood: 100%

Causal Attribution: Other

Coping Potential: moderate

Emotion: Anger(50)

Inhibits

Coping

ChallengeDesirability: 50

Likelihood: 50%

Causal Attribution: self

Coping Potential: Moderate

Emotion: Hope(25)

Future ActCause: self

Intend: yes

Probability: 50%

GoalUtility: 100

Probability: 50%Intend-that: True

Facilitates

Resignation

(abandon goal)

Modeling Appraisal & Coping

• Appraisal as evaluation of the causal interpretation– Define appraisal variables in terms of features of interpretation

– Fast, automatic

• Coping: Operators that suggest ways to change interpretation

– Sequential, deliberate, mediated by focus of attention

– Problem-focused Take Action, Make Plans

– Emotion-focused

• Denial/Wishful Thinking Change belief / likelihood

• Find silver lining Change utilities

• Shift blame Change causal attribution

(socially) Dialogue moves

• Distancing/acceptance Drop goal / intention

• Avoidance Change topic

Add goal (e.g., go to party)

Key: A “Content” & “Process” Model

Example: Modeling Naturalistic Emotion

Surprise Fear Anger

Empathy Humor/Relief

Emotional Dynamics

Relevance

Implication

Coping

Potential

Normative

Significance

“Working Memory”Environment

Control Signals

Appraisal

Frames

Affective

State

Appraisal

Coping

Inference Action

Appraisal

Scherer’s Sequential Checking Model EMA

Collaboration with Klaus Scherer’s Geneva Emotion Group

Relevance

Implication

Coping

Potential

Normative

Significance

“Working Memory”Environment

Control Signals

Appraisal

Frames

Affective

State

Appraisal

Coping

Inference Action

Appraisal

Scherer’s Sequential Checking Model EMA

Dynamics in

the world

Marsella & Gratch, JCSR 2008

Relevance

Implication

Coping

Potential

Normative

Significance

“Working Memory”Environment

Control Signals

Appraisal

Frames

Affective

State

Appraisal

Coping

Inference Action

Appraisal

Scherer’s Sequential Checking Model

Dynamics in

the world

Dynamics in perceived

world relationship

Marsella & Gratch, JCSR 2008

Relevance

Implication

Coping

Potential

Normative

Significance

“Working Memory”Environment

Control Signals

Appraisal

Frames

Affective

State

Appraisal

Coping

Inference Action

Appraisal

Scherer’s Sequential Checking Model

Dynamics in

the world

Dynamics in perceived

world relationship

Dynamics

through action

Marsella & Gratch, JCSR 2008

Bird Flies InPersonal HealthDesire: Self (100)

Satisfied: TrueProbability: 80%

Surprise

Perspective: Self

Unexpectedness: High

Controllability: Low

Blame/Credit: unresolved

Ap

pra

isal

Virtual Actor’s Appraisal of Bird Flying in Window

Seek

Info

Coping: Seek More Information

Bird Attack

Responsibility: Bird

Bird Flies In Personal HealthDesire: Self (100)Satisfied: FalseProbability: 60%

Fear: 48

Further Inference: The Bird will attack

Perspective: Self

Desirability: Undesirable

Controllability: Low

Blame/Credit: Bird

Ap

pra

isal

Threatens

Move Away

Avoid

Threat

Coping: Avoid

Bird Attacking

Responsibility: Bird

Bird Flies In

Personal HealthDesire: Self (100)Satisfied: FalseProbability: 50%

Anger: 48

Re-Appraisal & Further Planning

Perspective: Self

Desirability: Undesirable

Controllability: Medium

Blame/Credit: Bird

Ap

pra

isal

Threatens

Attack Threat

Whack Bird

Coping: Plan

Threatens

Bird at safer dx

Control Signals

Causal Interpretation(Goals, Beliefs, Causal Relations, Plans, Intentions)

Environment

AppraisalFrames

AffectiveState

Appraisal

Coping

Plan

Inference

Belief

Formation

Explanation

Action

Dialogue

Emotion as Meta-control system

Dynamics a function of coping/cognitive processes and evolving situation

48

This is the Seeley G. Mudd

Psychology building

How many minutes will it

take to climb the stairwell

to the top?

49

Mood induction

• Experimental manipulations can impact a person’s mood

– Happy/Sad Music

– Imagined situations

– Movies

– Weather

• These manipulations seem to impact unrelated decisions

– Sad music will make hills seem higher, math problems harder

– Performing experiment in disgusting room will increase sense

of moral outrage

– This has been called the “Misattribution” of emotion (Clore)

• People use feelings to inform whatever decision is in mind

– Cognitive heuristic?

50

Views on the influence between emotion and cognition

Elicitor

(e.g. Appraisal)

causes

Cognitive/Behavioral

Changes

Appraisal

Emotion

causes

Cog/Behav

Changes

Brain/Body

Configuration

causes

Feeling

Other factors Appraisal Other factorscauses

Cog Changes

Direct Influence Emotion as Mediator Emotion as Information

(Affect as information)

Zajonc