Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi –...
-
Upload
skyler-hammer -
Category
Documents
-
view
216 -
download
1
Transcript of Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi –...
![Page 1: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/1.jpg)
Towards Affective Interactive ECAs
Catherine Pelachaud – University of Paris 8
Isabella Poggi – University of Rome 3
WP6 workshop, 10-11 March 2005, Paris
![Page 2: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/2.jpg)
humaine WP6 workshop2
WP6 Emotion in Interaction
Embodied Conversational Agents
WP6 core task: describe an interactive ECA system with capabilities beyond those of present day ECAs
Conceptual framework consists of three key domains: perception, interaction and generation
Advance ECA capabilities in the key domains of perception, interaction and generation
![Page 3: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/3.jpg)
humaine WP6 workshop3
WP6 Emotion in Interaction
Perception: perception from audio and visual information;
Agent perception of the users / other agents / world.
human perceptual attention
Interaction: role of emotions in interaction; Balanced perception and generation
Interactants model of speaker(s) and listener(s)Agent Interpretation of discourse signals Active interpretation of signals
Social attention for speakers and listeners based on facial expressions during social interaction
Create affective awareness
![Page 4: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/4.jpg)
humaine WP6 workshop4
WP6 Emotion in Interaction
Generation: design of expressive signs. Produce dynamic expressive visual and auditory behaviours
Achieve coordination of signs of emotion in multiple modalities
Define representation language to encode information and use it to drive animation of ECAs
![Page 5: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/5.jpg)
humaine WP6 workshop5
Current conception
ECAs with more or less sophisticated presentation facilities. ECAs are to a certain extent able:
to compute what to say based on discourse plans
to generate synchronized behaviours
to talk with emotional appropriate intonation
to show some emotional expressions
to perform gesture and body movement
![Page 6: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/6.jpg)
humaine WP6 workshop6
What needs to be done
improve the naturalness of ECAs
multimodal integration and the display of emotional behaviours
improve the believability of ECAs
consistency of an ECA’s behaviour in terms of personality, cultural context and situation.
improve the interactivity of ECAs
take into account the particular user and the particular context
markes up its own model of user and of context
![Page 7: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/7.jpg)
humaine WP6 workshop7
ECA
ECA ought to be able to interact with the User not only through written text, but with all modalities:
words, voice, gesture,
gaze, facial expression,
body movements, body posture
(even through touch?)
conceive, represent and convey all the possible meanings that natural language and multimodal interaction convey in humans
![Page 8: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/8.jpg)
humaine WP6 workshop8
ECA - Capabilities
Capabilities required from an ECA corresponds to steps to move research forward
WP6 exemplar: Definition of Affective Interactive Embodied Conversational Agents with several capabilities covering the 3 domains:
perception
interaction
generation
![Page 9: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/9.jpg)
humaine WP6 workshop9
ECA - Capabilities
Cognitive influences on action Emotion related attention shifts
Adapt politeness behaviours to the user’s emotional state
Creating Affective Awareness Creating Affective Bonds
Imitation
Adaptation
Backchanneling
![Page 10: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/10.jpg)
humaine WP6 workshop10
ECA - Capabilities
Coordination of signs in multiple modalities From multimodal emotional corpora to models of coordination between modalities
Multimodal behaviour
Gesture repositories
Expressivity Behaviour expressivity
Speech Expressivity
Context dependent emotional body gesture
![Page 11: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/11.jpg)
humaine WP6 workshop11
Phases of Research
Empirical researchModelling Implementation Evaluation
![Page 12: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/12.jpg)
humaine WP6 workshop12
Phases of Research
Phase 1 – Empirical Research Find out regularities in mind and behavior of HumansDefine lexicons for all modalities: face, gaze, gesture, … through:
empirical research (questionnaires)analysis of observational data
Cooperation with WP3 (Theories and Models) and WP5 (Data and Database)
![Page 13: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/13.jpg)
humaine WP6 workshop13
Phases of Research
Phases 2 and 3 – Modelling and Implementation
formalisation and representation of the rules found out in the previous phase
Agent made of mind and bodyrepresent signals properties (shape and temporal dynamism)represent mind model
Cooperation with WP4 (Signs of Emotion), WP7 (Emotion in Cognition and Action), WP8 (Communication and Emotion)
![Page 14: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/14.jpg)
humaine WP6 workshop14
Phases of Research
Phase 4 – Evaluationsimilarity of ECA behavior with human behavior
believability of ECA animation
added-value of ECA in interaction system
how ECAs fit user’s need
Cooperation with WP9 (Usability)
![Page 15: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/15.jpg)
humaine WP6 workshop15
Example of ECA
Greta: ECA systemexpressive agent
individuality specification
based on a model of communicative functions by I. Poggi
![Page 16: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/16.jpg)
humaine WP6 workshop16
Model of Discourse and Conversation
Based on Castelfranchi & Parisi and Poggi
A sentence performs an action. It aims at a goal, and possibly at one or more supergoals
A discourse is made of several sentences which aims at the same supergoals
A conversation is a sequence of discourse where interactants “adopt” or “adhere” to each other’s goals
adopt: I1 acts to fulfil I2’s goal even if I2 has not asked I1 to do so
adhere: I1 acts to fulfil I2’s goal that I2 has explicitly asked I1 to do
![Page 17: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/17.jpg)
humaine WP6 workshop17
Multimodal Communication
Taxonomy of Communicative Functions (I. Poggi)
The speaker may provide three broad types of information about:
Information about the world: deictic, iconic (adjectival),…
Information about the speaker’s mind:
belief (certainty, meta cognitive)
Goal (performative, rheme/theme, turn-system, belief relation)
emotion
Information about speaker’s identity (sex, cutlure, age…)
![Page 18: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/18.jpg)
humaine WP6 workshop18
Information about Speaker’s Belief
Speaker has the goal that Addressee believes some specific speaker’s beliefs
Certainty markers: Indicate how reliable the provided information is
Meta-cognitive marker: Meta information about the source or cognitive states of information the speaker is taking about: gaze of thought.
Goal S Bel A (Bel S b)
![Page 19: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/19.jpg)
humaine WP6 workshop19
Information about Speaker’s Goal
Information about intention of single communicative act: performative of a sentence
Information about a whole hierarchy of intentions: planning of a sentence
Information about overall arrangement of conversation: turn-taking
Information about the relation between beliefs
Goal S Bel A Intend S Ag_Act
Ag_Act: agent’s action
![Page 20: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/20.jpg)
humaine WP6 workshop20
Performative Marker
Communicate illocutionary force of speech actI order you to put on your coat
I implore you to put on your coat
I advise you to put on your coat
Communicative intention of speaker that includes:
Her goals (her reason to communicate to the addressee, what she wants the addressee to do)
Her social relationship she wants to establish with addressee
![Page 21: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/21.jpg)
humaine WP6 workshop21
Performative
3 main classes of performative (General Goals):
request, inform, ask
Request: command, implore, advise…
Inform: warn, tell, announce…
Ask: question, interrogate…
Propositional content: set of beliefs speaker is mentioning and what she has to say about them.
![Page 22: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/22.jpg)
humaine WP6 workshop22
Performative
Elements distinguishing performatives within a same General Goal:
In whose interest is the action requested: S or A* Advise vs command, inform vs warn
Degree of certainty of S’s beliefs: certain, uncertain
* Suggest vs assure
Power relationship between S and A:* Command vs implore
![Page 23: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/23.jpg)
humaine WP6 workshop23
Performative: Peremptory Order
Example:In whose interest is the action: S
Interest S aDegree of certainty: unmarkedPower relationship: S has power on A
Bel S (Power-on S A)Performative: S orders A to do action a.
Ex: I order you to come here right awayGoal S (Do A a)
Signal of peremptory order: S looks down on A and frowns
Bel S (if (not Do A a) then (Angry S))
![Page 24: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/24.jpg)
humaine WP6 workshop24
Performative: Implore
Example:In whose interest is the action: S
(interest S a)Degree of certainty: unmarkedPower relationship: A has power on S
Bel S (Power-on A S)
Performative: S implores A to do action aEx: I implore for your help; I can’t do it otherwise
Goal S (Do A a)
Signal of imploration: S bends head aside and raised inner eyebrow
Bel S (if (not Do A a) then (Sad S))
![Page 25: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/25.jpg)
humaine WP6 workshop25
Information about Speaker’s Affective State
Emotion are triggered by an event, action, person
One can feel emotion toward another person (love, scorn)
Emotion triggered by an event and not directed toward someone (fear, surprise)
Signal: facial expressions of emotion
Display of emotion are regulated
Display or not of the emotion based on context
Goal S Bel A (Feel S e)
![Page 26: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/26.jpg)
humaine WP6 workshop26
Language of control of agent’s behaviors
Based on APML: Affective Presentation Markup
Language
XML-based mark-up language
Describe the communicative functions (I. Poggi)
Control the agent’s animation
Provide a mapping between meaning and signal
![Page 27: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/27.jpg)
humaine WP6 workshop27
Expression Meaning
deictic: this, that, here, there
adjectival: small, subtle, difficult,…; or big, long, great,…
certainty: certain, uncertain, …
metacognitive: I'm planning, I'm thinking, …
performative: greet, request (implore, order, suggest...), inform (warn, approve...),
topic comment: this is the topic, this is the comment
Belief relation: contrast,…
turn allocation: take turn, give turn
affective: anger, fear, happy, sorry-for, envy, relief, …
![Page 28: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/28.jpg)
humaine WP6 workshop28
Expression signals
Deictic: gaze direction, head direction
Certainty:Certain: small intensity frown
Uncertain: raised eyebrow
adjectival:Small, tiny…: small eye aperture, small opening of fingers
Belief relation: contrast: raised eyebrow; sequential symmetrical hand palm up
Performative: Suggest: small raised eyebrow, head aside
Emotion:Sorry-for: head aside, inner eyebrow up
…
![Page 29: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/29.jpg)
humaine WP6 workshop29
Example
<performative type="inform"><theme> As far as
<emphasis x-pitchaccent="LplusHstar"> vitamins </emphasis>
are concerned <boundary type="LH"/></theme>
</performative><rheme><emphasis x-pitchaccent="Hstar"> research </emphasis>
has shown <boundary type="LH"/></rheme>
<performative type="inform"><rheme>that eating
<emphasis x-pitchaccent="Hstar">the recommended </emphasis>levels of vitamin
<emphasis x-pitchaccent="Hstar"> A </emphasis>and
<emphasis x-pitchaccent="Hstar">C</emphasis><boundary type="LH"/></rheme>
<rheme affect="sorry-for">can have <emphasis x-pitchaccent="Hstar">beneficial </emphasis><emphasis x-pitchaccent="Hstar">effects </emphasis>for your
<emphasis x-pitchaccent="Hstar">appearance </emphasis> <boundary type="LH"/>
and <emphasis x-pitchaccent="Hstar">health</emphasis><boundary type="LL"/>
</rheme></performative>
![Page 30: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/30.jpg)
humaine WP6 workshop30
![Page 31: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/31.jpg)
humaine WP6 workshop31
Overall Architecture
Björn Hartmann, Myriam Lamolle, Maurizio Mancini
![Page 32: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/32.jpg)
humaine WP6 workshop32
Input
Input text marked with APML Communicative Function tags and their coefficient of communicative strength
Agent DefinitionExpressivity of agent
Spatial, temporal, fluidity, power, repetitivity, overallActivity
Agent-specific hierarchy of modalities (Which modalities the Agent tends to prefer?)
Face, gaze, gesture, posture, …
Predisposition factors (How expressive are each modalities?)Face, gaze, gesture, posture, …
Database of <meaning, signals, relevance>
![Page 33: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/33.jpg)
humaine WP6 workshop33
Video
![Page 34: Towards Affective Interactive ECAs Catherine Pelachaud – University of Paris 8 Isabella Poggi – University of Rome 3 WP6 workshop, 10-11 March 2005, Paris.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a78a75503463e778b62ba/html5/thumbnails/34.jpg)
humaine WP6 workshop34
Conclusion
Creation of an Affective Interactive Embodied Conversational Agent
perception
interaction
generation
Collaboration requires within and across Workpackges