Post on 24-Feb-2016
description
An Incremental Multimodal BML Realizer for Behavior Co-Articulation and Coordination
Herwin van WelbergenDennis Reidsma
Stefan Kopp
Beyond turn taking interaction◦ Continuous perception and behavior generation
Interpersonal coordination◦ Tight coordination of the (ongoing) behavior of
the agent with other (virtual) humans
Goal: Fluent interaction with virtual humans (and robots)
Acknowledge reception, understanding, attitude while listening
Respond to turn taking signals◦ Graceful interruption
Elicit backchannel feedback (and wait for it) Establish joint attention Exercise together with the interlocutor
◦ Synchronized movement Construct utterances incrementally, on the
fly◦ No long upfront planning
Fluent behavior: some examples
Between the retraction of one gesture and the preparation of the next◦ Retraction of the first gesture is skipped
Co-articulation (or lack thereof) can have a communicative function (Kendon 1980)◦ E.g. marking information boundaries
Has to be established on the fly
Gesture co-articulation
Allow last minute changes to ongoing behavior◦ Top down (through BML) ◦ Bottom up emergent behavior
But don’t break BML constraints Real-time, incremental, fluently connect BML
increments Under-specification handling
◦ The realizer should be smart enough to figure out unconstrained timing, generate co-articulation, ..
◦ This should happen ‘on the fly’◦ As late as possible
Behavior realizer requirements
A BML block specifies some constraints in shape and timing on its behaviors
BML blocks are generally underspecified◦ The BML Realizer has certain realization freedom◦ Used to achieve natural looking motor behavior
BML
BML
Example
Offers specification for multimodal synchronization, but:◦ No specification mechanisms for inter-personal
synchronization◦ Limited specification capabilities for
incrementality◦ No specification for co-articulation
BML
SmartBody (Thiebeax et al 2008) EMBR (Heloir and Kipp 2010) Greta (Mancini et al 2008) RealActor (Čerecović and Pandžić 2010) Special focus on incrementality/interactional
coordination◦ Elckerlyc (van Welbergen et al. 2010)◦ ACE (Kopp and Wachsmut 2004)
BML Realizers
Realizing BML the traditional way
Flexible behavior plan representation (Reidsma et al. 2011)◦ retains BML constraints
Modified in top-down fashion
Specification mechanisms for interactional coordination
Elckerlyc
Incremental behavior planning
Specification mechanism for chunk composition
Emergent gesture co-articulation through bottom-up plan modifications
ACE
Co-articulation can have a communicative function
The Behavior Planner should be able to specify whether or not co-articulation may occur
Results: gesture co-articulation
<bml id="bml2" composition="append-after(bml1)"> <speech id="speech1"> <text> At <sync id="s1"/> 6 pm you have another appointment. </text> </speech> <gesture id="gesture1" lexeme="BEAT" stroke="speech1:s1"/></bml>
Specifying composition
<bml id="bml2" composition="chunk-after(bml1)"> <speech id="speech1"> <text> At <sync id="s1"/>6 pm you have another appointment. </text> </speech> <gesture id="gesture1" lexeme="BEAT" stroke="speech1:s1"/></bml>
Start a new BML block when a previous block is ‘retracting’◦ All behaviors have a retraction phase◦ A BML block is ‘retracting’
If all its behaviors are either finished or retracting◦ If needed, fluently overtake the retraction phase
of a gesture in the previous block
Implementing gesture co-articulation
No co-articulation
Co-articulation
Initial plan
Top down replanning step
Bottom-up adaptation step
Gracious interruption depends on◦ Behavior state (e.g. not started, retracting)◦ Resting posture/state◦ Current position of e.g. the hand
This information is available in the AnimationEngine◦ Smart retraction using the AnimationEngine
Replace a interrupted gesture by a new one◦ Co-articulation
Results: gracious interruption
Interruption
Top down specification that a pointing gesture/gaze should remain on target until an interlocutor event◦ Joint attention◦ Feedback elicitation
Bottom up emergent insertion/deletion of ‘hold’ motions
Usage scenario: interactional coordination
AsapRealizer combines state of the art features of ACE and Elckerlyc
Fusion provides new capabilities:◦ Interactional coordination + automatic hold phase
construction◦ Top down interrupt with bottom-up automatic
graciousness + co-articulation Top down+bottom up adaptation
capabilities are essential for fluent interaction
Summary
Questions?
http://asap-project.org
Thanks for your attention
SAIBA
Behavior/block state machine
Internal timing predictions of one or more modalities might be unpredictable◦ E.g. for robot behavior◦ Bottom up adjustments can be used to
Adjust the behavior itself (e.g. speed up to meet a time constraint)
Move the time constraints that cannot be met, automatically adjusting the timing on other modalities linked to these constraints
Usage scenario: robotics
Interruption in ElckerlycResults: Interruption
(a) Interrupting all behavior in bml1.<bml id="yieldturn"><bmlt:interrupt id="i1" target="bml1"/></bml>(b) Interrupt all behavior in bml1 excluding gesture1.<bml id="yieldturn"><bmlt:interrupt id="i1" target="bml1" exclude="gesture1"/></bml>(c) Interrupt all behavior in bml1. Insert a behavior (relaxArm) that gracefully moves the gesturing arm back to its rest position.<bml id="yieldturn"><bmlt:interrupt id="i1" target="bml1"/><bmlt:controller id="relaxArm" class="CompoundController" name="leftarmhang"/></bml>
Gesture is specified in a TimedMotionUnit◦ TimedMotionUnit specification:
State Priority Set of controlled joints
◦ TimedMotionUnit execution: Set a joint rotation on the skeleton Add a physical controller Set a RestingTimedMotionUnit
AsapAnimationEngine
RestingTimedMotionUnit◦ Manages motion related to the resting state of the
virtual human Possible implementations:
Static posture (as in ACE) Lower body physical balance controller …
◦ Is always executed with the lowest priority◦ Can create transition TMUs to the resting state
Slerp Drop the arm controller …
AsapAnimationEngine
The priority of a TMU drops in its SUBSIDING state
AsapAnimationEngine executes TMUs in order of priority
If a TMU needs to execute motion on joints that are already in use by a higher priority TMU, drop the TMU
Automatically create a cleanup TMU that moves other joints back to the resting state
Gesture Coarticulation in the AsapAnimationEngine
Gesture changes during execution:◦ start position of the hand◦ hand position at the start and end of the stroke ◦ posture state at the end of the gesture
Adaptive timing and shape for gesture preparation and retraction phase◦ Using Fitts’ law◦ Only update if no other constraints act on e.g.
both start and stroke-start timing
TimedMotionUnit bottom up adaptations
BML specification mechanisms for:◦ Interpersonal coordination
Gracious interruption [but verbose and top-down only]
Flexible plan representation◦ Allows adaptation of the ongoing plan◦ While maintaining BML constraints◦ So far such plan modifications were mainly
managed top-down (e.g. by new BML blocks)◦ So far no combined shape and timing adaption
mechanisms
Elckerlyc
MURML specification for multimodal coordination
Incrementality◦ Gesture co-articulation
Bottom up adaptation of ongoing behavior
ACE
Top-down specification of a parameter change◦ Increase the amplitude of an ongoing gesture
Bottom up adaptations:◦ Update retraction and/or preparation timing and
shape accordingly◦ Might adjust unconstrained shape and timing
parameters to retain biological plausibility
Usage scenario: parameter change
Gesture-speech alignment
Real-time multimodal behavior generation for robots and virtual humans◦ Incremental construction of behavior
Fluent connection of increments (e.g. behavior co-articulation)
◦ Allowing last minute changes of ongoing behavior◦ Multimodal synchronization
Builds upon ACE, Elckerlyc BML
◦ BML 1.0 compliant (tested by RealizerTester)
AsapRealizer