Interacting with Virtual Humans Zsófia Ruttkay Associate Professor Human Media Interaction, Dept....

Interacting with Virtual HumansInteracting with Virtual Humans

Zsófia RuttkayZsófia Ruttkay

Associate ProfessorAssociate Professor Human Media Interaction, Dept. of CS, University of Human Media Interaction, Dept. of CS, University of

Twente, NLTwente, NLhttp://hmi.ewi.utwente.nl/~zsofi/http://hmi.ewi.utwente.nl/~zsofi/

SSIP 2009 DebrecenSSIP 2009 Debrecen

[email protected]

http://hmi.ewi.utwente.nl/~zsofi/

2

Overview of the talkOverview of the talk

FAFA Analysis of facial expressions of humans and Analysis of facial expressions of humans and cartoonscartoons

Emotion DiscEmotion Disc

Challenges for IPChallenges for IP

Non-photorealistic facial animation - Non-photorealistic facial animation - CharToonCharToon

Full bodyFull body Control with markup – GESTYLEControl with markup – GESTYLECurrent work - conductorCurrent work - conductor

IntroIntro About Virtual HumansAbout Virtual Humans

Evaluation of personality Evaluation of personality cuescues

3

About Virtual Humans IAbout Virtual Humans I

• More or less human-like synthetic characters to interact More or less human-like synthetic characters to interact

with userswith users

• In roles as:In roles as:

- information provider- information provider

- tutor- tutor

- sales assistant- sales assistant

- ‘translator’ (sign language)- ‘translator’ (sign language)

- avatar in telepresence and VR- avatar in telepresence and VR

- entertainment, games- entertainment, games

- also medium for experiments- also medium for experiments

• Research since 15 years, recently own events:Research since 15 years, recently own events:

IVA, Gesture WS, AAMAS workshop seriesIVA, Gesture WS, AAMAS workshop series

• Book by J. Cassell et al. (2000, MIT Press)Book by J. Cassell et al. (2000, MIT Press)

4

ExamplesExamplesRea from MIT, Cassell et al.

Steve from USC J. Rickel, L. Johnson et J. Rickel, L. Johnson et al.al.

5

ExamplesExamples

OLGA – KTH

6

ExamplesExamples

Max – Univ. of Bielefeld

7

ExamplesExamples

Mission Exercise, USC ISI + ICT

8

ExamplesExamples

Carmen’s Bright Ideas from USC ICT, Marsella et al.

9

ExamplesExamples

USC – ISI, Adele, Shaw et al.

10

ExamplesExamples

Let’s Face It! –M. Bartlett et al. Univ. San Diego, 2008.

11

Virtual conductorVirtual conductor

Virtual Conductor HMI, Univ. of Twente, The Netherlands

12

Virtual scientific instituteVirtual scientific institute

Meta Institute for Computational Astrophysics

Piet Huthttp://www.ugotrade.com/2008/07/19/astrophysics-in-virtual-worlds-implementing-n-body-simulations-in-opensim/

13

RobotRobotss

Hiroshi Ishiguro ATR Intelligent Robotics and Communications Laboratorieshttp://www.irc.atr.jp/Geminoid/

14

GGeminoidseminoids

Geminoid by I. Ishiguro at ATR, Japanhttp://www.irc.atr.jp/Geminoid/

15

decision(what to do)

About VHs -About VHs - general general architecturearchitecture

perception (input)

action (output)

• nl processing• (meta) speech understanding• face interpretation• gesture recognition• environment monitoring (object, resources, situation)

• multimodal output• related to environment• user model

• affect model• discourse model• domain knowledge• goals …

• real-time• suspension of disbelief• ‘good’ for …

16

About VHs – Design About VHs – Design parametersparameters

• Embodiment: head – torzo - full Embodiment: head – torzo - full

• ‘‘Look’: gender, age, profession, personalityLook’: gender, age, profession, personality

• Design: realistic/not, 2d/3dDesign: realistic/not, 2d/3d

• Communication channels & directionsCommunication channels & directions

speech, facial expressions, gestures, body languagespeech, facial expressions, gestures, body language

• What is to be expressed: emotional, cognitive, phys. What is to be expressed: emotional, cognitive, phys.

state state

references to changing surrounding (incl. user)references to changing surrounding (incl. user)

• Role: monologue – dialogue, Role: monologue – dialogue,

relation to user: gender, cluture, socal status relation to user: gender, cluture, socal status

17

About VHs – Hierarchy of About VHs – Hierarchy of behaviorbehavior

• independent behaviorindependent behavior• goal-oriented goal-oriented behaviorbehavior• personality, emotionspersonality, emotions• high-level controlhigh-level control• bidirectional mm bidirectional mm comm.comm.• multimodal outputmultimodal output• single modal outputsingle modal output• rendering, TTS, ...rendering, TTS, ...

18

About VHs - development About VHs - development cyclecycle

analysis of H-H interaction

computationalmodel

implementation of VA

evaluationof VA

non-realism:animation,

art

What? How?

realtime adoptabl

e

micro levelin

application

mindbody

19

About VHs About VHs – Some issues– Some issues

• Mind and bodyMind and body issue issue

• Why to useWhy to use VHs? VHs?

- ease of use: broader user group, less load- ease of use: broader user group, less load

- more effect- more effect

- more fun- more fun

• Moral and legalMoral and legal issues: clones, issues: clones,

manipulationmanipulation

• To To mimic humansmimic humans, or less is more?, or less is more?

• InterdisciplinaryInterdisciplinary research research

20

About VHs About VHs – Do people take – Do people take themthem??

• IllusionIllusion – do people believe, react to as if – do people believe, react to as if

theythey

were real?were real?

• YES YES

Clifford Nass: CASA paradigm Clifford Nass: CASA paradigm

Computers Are Social ActorsComputers Are Social Actors

• Series of experiments on effect of Series of experiments on effect of

micro and macro designmicro and macro design

• User’s characteristicsUser’s characteristics matter matter

21









22

Motivations for FA researchMotivations for FA research• Need for: Need for: expressiveexpressive, animated synthetic faces, animated synthetic faces

• FunctionsFunctions of facial (and other) nonverbal signals: of facial (and other) nonverbal signals: - emotions - emotions - cognitive state- cognitive state

- speech punctuation- speech punctuation- dialog control- dialog control- additional/redundant info (size, location, …)- additional/redundant info (size, location, …)

- personality and culture of speaker- personality and culture of speaker

• All contribute to: comprehension, memory, effectivity All contribute to: comprehension, memory, effectivity … … motivation, joy, engagement motivation, joy, engagement

23

Questions concerning FAQuestions concerning FA

• Which Which expressionsexpressions to consider? to consider?

• What are those expressions like?What are those expressions like?

- - static and dynamicalstatic and dynamical characteristics characteristics- - generic and individualgeneric and individual characteristics characteristics

• What What facial modelfacial model to use? to use? - (non-)photorealism- (non-)photorealism- beyond realism/simplification of motions?- beyond realism/simplification of motions?

• Tool to make synthetic face models andTool to make synthetic face models and animate them animate them

24

Facial anatomyFacial anatomy

• about 44 pairs of musclesabout 44 pairs of muscles

– bundlebundle

– sheetsheet

– circularcircular

• physiological data physiological data missingmissing

• coarticulation coarticulation

• many parametersmany parameters

• nonlinear effectsnonlinear effects

25

Psychology of facial expressionsPsychology of facial expressions

• ““meaningful” – physically possible - impossiblemeaningful” – physically possible - impossible

• biological need/content/emotional state/discourse biological need/content/emotional state/discourse regulator/…regulator/…

• muscle activity – emotion sensing in brainmuscle activity – emotion sensing in brain

• difference between real and ‘on demand’ difference between real and ‘on demand’ expressions expressions

data collection must be done with care:data collection must be done with care:- spontaneous/stimulated expression - spontaneous/stimulated expression - natural setting (e.g. in news)- natural setting (e.g. in news)- real people vs. actor?- real people vs. actor?

26

History ….History ….

• 1882: 1882: Darwin: The Expression of the Darwin: The Expression of the Emotions in Man and AnimalsEmotions in Man and Animals

• 1862: 1862: Duchenne: Duchenne: photos of expressions photos of expressions induced by electric signals induced by electric signals

28

HistoryHistory: coding of facial : coding of facial expressionsexpressions

• 1970 - ...: Paul Ekman (U. of California)1970 - ...: Paul Ekman (U. of California)

• 1989: 6 1989: 6 general emotional expressionsgeneral emotional expressions ::happiness, surprise, happiness, surprise, sadness, fear, disgust, angersadness, fear, disgust, anger

• FACS [1978] coding systemFACS [1978] coding system – – still preferred in still preferred in USUS

• discretediscrete

• visual effectof (groups of) musclesvisual effectof (groups of) muscles

29

Coding of facial expressionsCoding of facial expressions: : MPEG4 MPEG4

• 1999: ISO standard1999: ISO standard

• widely usedwidely used

• 6868 normalised parameters normalised parameters ((FAPFAPs)s)

• continuouscontinuous

• black points x, y, (z) displblack points x, y, (z) displ

• min, max, min, max, neutral valueneutral value

30

Emberi arckifejezEmberi arckifejezésekések leirása leirása: : MPEG4 MPEG4

• # FAP name # FAP name FAP description FAP description UnitUnit

• 31 31 raise_l_i_eyebrow raise_l_i_eyebrow Vertical displacement of left inner eyebrow Vertical displacement of left inner eyebrow ENSENS

• 32 32 raise_r_i_eyebrow raise_r_i_eyebrow Vertical displacement of right inner eyebrow Vertical displacement of right inner eyebrow ENSENS

• 33 33 raise_l_m_eyebrowraise_l_m_eyebrow Vertical displacement of left middle eyebrow Vertical displacement of left middle eyebrow ENSENS

• 34 34 raise_r_m_eyebrow raise_r_m_eyebrow Vertical displacement of right middle eyebrow Vertical displacement of right middle eyebrow ENSENS

• 35 35 raise_l_o_eyebrow raise_l_o_eyebrow Vertical displacement of left outer eyebrow Vertical displacement of left outer eyebrow ENSENS

• 36 36 raise_r_o_eyebrow raise_r_o_eyebrow Vertical displacement of right outer eyebrow Vertical displacement of right outer eyebrow ENSENS

• 37 37 squeeze_l_eyebrow squeeze_l_eyebrow Horizontal displacement of left eyebrow Horizontal displacement of left eyebrow ESES

31

Analysis of facial expressions Analysis of facial expressions – own investigation 2002– own investigation 2002• On demand expressions of untrained subjectsOn demand expressions of untrained subjects

• Track 15 FAPs marked on faceTrack 15 FAPs marked on face

32

The E-Space: PCAThe E-Space: PCA

FAP variables component 1component 2component 347.6% 26.3% 9.41%

3 open mouth -0.17 0.22 0.634 lower middle upper lip -0.20 0.25 -0.315 raise middle lower lip 0.17 -0.26 -0.596 raise left corner point mouth 0.12 -0.34 -0.047 raise right corner point mouth 0.14 -0.39 0.1312 stretch left corner point mouth 0.15 -0.41 0.1913 stretch right corner point mouth 0.15 -0.41 0.2031 raise left inner eyebrow -0.34 -0.12 -0.0432 raise right inner eyebrow -0.34 -0.09 -0.0233 raise left middle eyebrow -0.35 -0.14 -0.0534 raise right middle eyebrow -0.35 -0.14 -0.0235 raise left outer eyebrow -0.35 -0.07 -0.1336 raise right outer eyebrow -0.35 -0.10 -0.0637 squeeze left eyebrow 0.22 0.30 -0.1938 squeeze right eyebrow 0.23 0.23 0.02

33

The E-Space: 2 components’ The E-Space: 2 components’ graph graph (tracked)(tracked)

Hor: eyebrowsHor: eyebrows

Vert: mouth cornersVert: mouth corners

34

The E-Space: CharToon dataThe E-Space: CharToon data

35

The E-Space: CharToon dataThe E-Space: CharToon data

• Experienced animatorExperienced animator

• 59 expression stills:59 expression stills:– Anger: Be_Careful_What_You_SayAnger: Be_Careful_What_You_Say

– Smile: Absolute_Joy, Adoration, SmileSmile: Absolute_Joy, Adoration, Smile

– Other: Doubt_suspicionOther: Doubt_suspicion

36

The E-Space: Graph The E-Space: Graph (cartoon)(cartoon)

• Hor: eyebrowsHor: eyebrows

• Vert: stretch mouthVert: stretch mouth

• 3rd: smile/squeeze3rd: smile/squeeze

(all a-symmetric)(all a-symmetric)

37

The E-Space: Conclusion The E-Space: Conclusion (stills)(stills)

• Tracker FAPs not always sufficient to Tracker FAPs not always sufficient to distinguish (negative) emotionsdistinguish (negative) emotions

• More symmetry in tracked expressionsMore symmetry in tracked expressions

• Animator uses more of the expression Animator uses more of the expression spacespace

• smilingsmiling shape of the mouth and shape of the mouth and raisingraising of the eyebrows are most significantof the eyebrows are most significant

38

Time curves of FAPsTime curves of FAPs

39

Actuation of muscles during smile and anger (I. Essa, PhD at MIT, 1995)

40

The E-Space: Time curvesThe E-Space: Time curves

41

Co-actuation of muscles during smile (I. Essa, PhD at MIT, 1995)

42

Facial expression analysis Facial expression analysis - dynamism- dynamism

• Time IS characteristic and importantTime IS characteristic and important

• Trapezoid activation function, variants (camel, …) Trapezoid activation function, variants (camel, …)

• Still open questions: shapes, constraints, …Still open questions: shapes, constraints, …

• No data(base) published (MS either)No data(base) published (MS either)

43

STAR on facial expression STAR on facial expression trackingtracking Near real-time tracking of fapsNear real-time tracking of fapsOther than basic 6 expressionsOther than basic 6 expressionsMixture, masking, cultural variantsMixture, masking, cultural variantsDynamism of articulationDynamism of articulationSpontaneous expressionsSpontaneous expressions• Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek,C., Fasel, I., Movellan, J.R. Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek,C., Fasel, I., Movellan, J.R.

(2006). Automatic Recognition of Facial Actions in Spontaneous Expressions. Journal of (2006). Automatic Recognition of Facial Actions in Spontaneous Expressions. Journal of

Multimedia 1(6) p. 22-35. Multimedia 1(6) p. 22-35.

• Cohn, J. F. & Ekman, P. Methods for measuring facial actions. In J.A. Harrigan, R. Cohn, J. F. & Ekman, P. Methods for measuring facial actions. In J.A. Harrigan, R.

Rosenthal, & K. Scherer (Eds.), Handbook of nonverbal behavior research methods in Rosenthal, & K. Scherer (Eds.), Handbook of nonverbal behavior research methods in

the affective sciences. NY: Oxford.the affective sciences. NY: Oxford.

• Tian, Y.L., Kanade, T., & Cohn, J.F. Facial expression analysis. In S.Z. Li & A.K. Jain Tian, Y.L., Kanade, T., & Cohn, J.F. Facial expression analysis. In S.Z. Li & A.K. Jain

(Eds.), Handbook of face recognition. NY: Springer. (Eds.), Handbook of face recognition. NY: Springer.

• KTH, Sweeden – cheap and light hw for facial trackingKTH, Sweeden – cheap and light hw for facial tracking

Analysis of Mona Lisa Analysis of Mona Lisa

smile: 83%disgust: 9% fear: 6%anger: 2%

result of computer facial expression analysis UvA + Univ. of Illionis, Dec. 2005

What do the numbers mean?How were they obtained?Would it mean the same in the 1500s?

45









46

Facial animation – model Facial animation – model basedbased

Basic principle:Basic principle:

• Deformation principles – 2d skeleton, 3d real/virtual Deformation principles – 2d skeleton, 3d real/virtual muscle, …muscle, …

• Key positions: values of controllersKey positions: values of controllers

• Inbetweens by interpolationInbetweens by interpolation

Issues:Issues:

• Reusable, parametrised repertoireReusable, parametrised repertoire

• BlendingBlending

• Superposition (talking while smiling)Superposition (talking while smiling)

• Visual speech - coarticulationVisual speech - coarticulation

47

Facial animationFacial animation• how to interpolatehow to interpolate: : linearlinear, C, C11, C, C22

• how to choose how to choose ttii (times for keyframes) (times for keyframes)

t1 t2 t3 t4 t5to

48

3D 3D modelmodel

49

3d facial modelling (CWI)3d facial modelling (CWI)

50

3d facial animation3d facial animation

51

MPEG4 defined animation for MPEG4 defined animation for different facesdifferent faces

52

Visual speechVisual speech

53

Lip-syncLip-sync

54

Some 2d head modelsSome 2d head models

55

Some movies made with Some movies made with CharToonCharToonby A. Lelievre and Zs. Paalby A. Lelievre and Zs. Paal

Lily

Frans

Nine faces

Magician

56

Personality and culture in facial Personality and culture in facial designdesign

57

3D realistic3D realistic vs. vs. 2D cartoon2D cartoon facesfaces

2D cartoon 3D realistic

model making easy very difficult

animation fast slow

aesthetics appealing disturbing

expressiveaspects

presenceemotionsspeech

anatomydetails

expressions beyondrealism!

withinrealism

58

Component repertoireComponent repertoire

Unlimited freedom Unlimited freedom • design faces from CharToon componentsdesign faces from CharToon components• finetune dynamismfinetune dynamism

Repertoire of: Repertoire of: • facial feature componentsfacial feature components• (still) expressions and animations(still) expressions and animations

Support to reuse the components:Support to reuse the components:• technical level (design, UI)technical level (design, UI)• ‘ ‘mix & match’ recipes with respect to expected mix & match’ recipes with respect to expected functionalityfunctionality

59

Animating faces in CharToon Animating faces in CharToon II

60

Animating faces in CharToon Animating faces in CharToon IIII

61

Animating faces in CharToon Animating faces in CharToon IIIIII+ constraints technology

62


63


64

Emotion DiscEmotion Disc• Cyclic neighborhood Cyclic neighborhood of 6 basic emotions of 6 basic emotions

• Tool to: Tool to:

- - exploreexplore the expression space the expression space

- - controlcontrol the expressions the expressions

• Experiments with expressive 2D faces Experiments with expressive 2D faces

-- how expressive they are?how expressive they are?

- contradicting expressions (2-3 discs) - contradicting expressions (2-3 discs)

65

From: Harold Schlosberg: The Description of Facial Expressions in Terms of Two Dimensions, Journal of Experimental Psychology, Vol. 44. No. 4. Oct. 1952, pp 229-237.

66









67

Evaluation of talking headsEvaluation of talking heads• Work with E. Krahmer at als. Work with E. Krahmer at als.

• Effect of facial signals to express: Effect of facial signals to express:

- emphasis --- eyebrow & pitch- emphasis --- eyebrow & pitch- personality --- eyebrow, gaze and speech- personality --- eyebrow, gaze and speech

• See publications, experimental mat. on web pageSee publications, experimental mat. on web page

68

The evaluation research The evaluation research questionsquestions

• Do facial cues contribute to the perception of Do facial cues contribute to the perception of

extraversion?extraversion?

• How do gaze + brows + speech cues add up?How do gaze + brows + speech cues add up?

– inconsistencyinconsistency

69

Personality for VHs Personality for VHs

• Personality is important:Personality is important:

– to improve comprehension (Laurel 1993)to improve comprehension (Laurel 1993)

– to like and trust and … the VA betterto like and trust and … the VA better

(Nass and Lee 2000, Isbister and Nass 2000, …) (Nass and Lee 2000, Isbister and Nass 2000, …)

– people attribute personality to agents people attribute personality to agents

CASA paradigm (Reeves and Nass 1996)CASA paradigm (Reeves and Nass 1996)

– fun, varietyfun, variety

70

Personality for VHsPersonality for VHs• Trait theory (e.g. Allport 1937)Trait theory (e.g. Allport 1937)

• ““Big 5” (e.g. Wiggins 1997): Big 5” (e.g. Wiggins 1997):

Extroversion (dominance),Extroversion (dominance),

Agreeableness, Conscientiousness, Agreeableness, Conscientiousness,

Emotional stability, CultureEmotional stability, Culture

• Cues studied so far:Cues studied so far:

– speech (pitch and duration)speech (pitch and duration)

– gazegaze

– posture (Isbister and Nass 2000)posture (Isbister and Nass 2000)

– languagelanguage

71

The experimentThe experiment

GazeGaze SpeechSpeech BrowsBrows

ExtrovertExtrovert

fixed on fixed on useruser

2 blink2 blink

wide pitch wide pitch rangerange

more variationmore variation

higher tempohigher tempo

2x accent 2x accent

3x100 ms 3x100 ms stages stages eacheach

IntrovertIntrovert

““move S-Wmove S-W

look aside look aside move move center”center”

2x2x

low pitch rangelow pitch range

little variationlittle variation

20% slower20% slower

no no movementmovement

2 x 2 x 2 = 8 personality profiles

72

The experimentThe experiment• Examples of stimuli from Examples of stimuli from http://http://

www.cwi.nl/~zsofi/stim.htmlwww.cwi.nl/~zsofi/stim.html

is_ig_ib es_eg_ibis_ig_ib

http://www.cwi.nl/~zsofi/stim.html

http://www.cwi.nl/~zsofi/stim.html

73

Evaluation of VHsEvaluation of VHs• Hot topic, evaluation is a MUSTHot topic, evaluation is a MUST

• Zs. Ruttkay, C. Pelachaud: From Brows to TrustZs. Ruttkay, C. Pelachaud: From Brows to Trust

(Kluwer, 2004)(Kluwer, 2004)

74









75

Research questions on Research questions on gesturesgestures

• What gestures do people use?What gestures do people use?- references to environment & user- references to environment & user (location, size, time, motion, shape,…) (location, size, time, motion, shape,…)- to indicate actions and concepts- to indicate actions and concepts- emotions, cognitive and physical state- emotions, cognitive and physical state- dialog status - dialog status

• What gestures should an VA use?What gestures should an VA use?- function - application domain- function - application domain- redundancy - expressivity- redundancy - expressivity

- realization – non-repetitive, tuned- realization – non-repetitive, tuned

• Modelling and sw engineering issues: Modelling and sw engineering issues:

motion characteristics, motion characteristics,

performance, reusability, …performance, reusability, …

As humans

?

YES, but …

76

Communicative Communicative functionsfunctions

emphasizeemphasize

take turn/take turn/start talkingstart talking

a) raise voicea) raise voiceb) look at listenerb) look at listenerc) raise eyebrowc) raise eyebrowd) beat with handd) beat with hande) 11 (?) combinationse) 11 (?) combinations

a) look at partnera) look at partnerb) raise handb) raise handc) voice signal (hm-c) voice signal (hm-

hm)hm)d) …d) …

Multi-modal Multi-modal signalssignals

77

Why to bother about style?Why to bother about style?

• Style is source of information on:Style is source of information on:

– gender, age, physical stategender, age, physical state

– personality, emotional statepersonality, emotional state

– ethnicity, professionethnicity, profession

– relationship between speaker and listenerrelationship between speaker and listener

– situation (public/private, …)situation (public/private, …)

……

• variety and joyvariety and joy

• does matter in HCIdoes matter in HCI (Nass et al. 2000, …) (Nass et al. 2000, …)

78

Our objectivesOur objectives

• VA with style in VA with style in

nonverbal communication (and speech)nonverbal communication (and speech)

• DeclareDeclare style of VA style of VA

• GenerateGenerate styled conversation by VHs styled conversation by VHs

• Framework (can be tailored/extended)Framework (can be tailored/extended)

• Easy to use mark-up languageEasy to use mark-up language

• Experiments!Experiments!

79

GESTYLE: the big pictureGESTYLE: the big picture

StyleDeclaratio

n

Meaning Tags

Combine Style Dictionaries into a single

one

Map Meanings toGesture

Expressions

Text with Gesture

Expressions (relative

time)

Expand to Basic Gestures with absolute timing

Text with Basic

Gesture Tags(abs. time)

CombinedStyle Dictionary

Timing of speech

Generate animationin rendering system

Gesture Dictionary

Text with Modifier

TagsModify Combined Style Dictionary

Modified CombinedStyle Dictionary

Style Dictionary 1Meaning mapping 1…Meaning mapping m1

Style Dictionary 2Meaning mapping 12

…Meaning mapping m2

…

Style Dictionary nMeaning mapping 1n

…Meaning mapping mn

Modifier Tags

input text with GESTYLE markups

80

GESTYLE: Style dictionaryGESTYLE: Style dictionary<StyleDictionary Name = "<StyleDictionary Name = "extrovertextrovert">">

<Meaning Name = "<Meaning Name = "emphasizeemphasize" CombinationMode = "DOMINANT">" CombinationMode = "DOMINANT">

<GestureSpec><GestureSpec>

<MannerDefinition intensity="<MannerDefinition intensity="intenseintense"/>"/>

<UseGest Name=“<UseGest Name=“NNododAndBAndBeateat"/><PAR>"/><PAR>

<UseGest Name=”<UseGest Name=”LookAtPersonLookAtPerson”/>”/>

<Probability P="<Probability P="0.70.7"/>"/>

</GestureSpec></GestureSpec>

<GestureSpec><GestureSpec>

<MannerDefinition motion_manner="<MannerDefinition motion_manner="sudden_onsudden_on"/>"/>

<UseGest Name=“<UseGest Name=“BBeateat"/>"/>

<Probability P="<Probability P="0.30.3"/>"/>

</GestureSpec></GestureSpec>

</Meaning></Meaning>

……

81

Definition and generation of Definition and generation of gesturesgestures• structural characteristicsstructural characteristics::

composed of basic gestures by composed of basic gestures by papar, r, seq seq and and repeatrepeat operatorsoperators

• performance (manner) characteristics:performance (manner) characteristics:

intensity, duration, start_time, precision, intensity, duration, start_time, precision, jerkyness, symmetryjerkyness, symmetry, … parameters, … parameters

82

GESTYLE: Combining stylesGESTYLE: Combining styles• Mechanism to resolve Mechanism to resolve conflicting prescriptionsconflicting prescriptions on:on:

– gestures gestures

– modalitiesmodalities

– mannermanner

• PreferencesPreferences to be given in Style Declaration to be given in Style Declaration

• Dynamical modificationsDynamical modifications of: of:

– modality usagemodality usage

– mannermanner

– style dominancestyle dominance

83

GESTYLE: Gesture repertoireGESTYLE: Gesture repertoire

<GestureRepertoire><GestureRepertoire>

<DefGest Name="<DefGest Name="NodNod" GestureLength = "200"/>" GestureLength = "200"/>

<DefGest Name="<DefGest Name="BeatBeat" GestureLength = "500" " GestureLength = "500" Symmetry=”right”/>Symmetry=”right”/>

......

<DefGest Name="<DefGest Name="NodAndBeatNodAndBeat" gesture_length = "500">" gesture_length = "500">

<UseGest Name="Beat" sub_start_time = "0" /><PAR/><UseGest Name="Beat" sub_start_time = "0" /><PAR/>

<UseGest Name="Nod" sub_start_time=”100”/></DefGest><UseGest Name="Nod" sub_start_time=”100”/></DefGest>

</GestureRepertoire></GestureRepertoire>

84

Two examlesTwo examles

• Hamlet:Hamlet: CharToon 2D CharToon 2D

– face, eyes, handsface, eyes, hands

– .avi.avi

• Yt:Yt: 3D H-anim humanoid for the Web 3D H-anim humanoid for the Web

– hand gestures onlyhand gestures only

– real-time, on the fly generation of animationreal-time, on the fly generation of animation

• Speech by MindMaker’s FlexVoice TTS, Speech by MindMaker’s FlexVoice TTS,

styled too for Hamlet!styled too for Hamlet!

85

HamletHamlet

HamletMarkUp1.xmlHamletMarkUp1.xml HamletMarkUp2.xmlHamletMarkUp2.xml

StyleDictionaries.xml

86

YtYt hierarchical articulated body, 54 DOF of hand joints

modular gesture definition written in STEP (VU)

emphasisemphasis enumerateenumerate sorry_forsorry_for greetinggreeting

87

Cognition and action Cognition and action The Functional Behaviour The Functional Behaviour Markup Languages initiativeMarkup Languages initiative

IntentPlanningIntent

PlanningBehaviorPlanningBehaviorPlanning

BehaviorRealizationBehavior

Realization

BehaviorRepository

(BML)

BehaviorRepository

(BML)

FML BML

88

What is BML? - AbstractionWhat is BML? - Abstraction

• gesture• head• speech• posture• gaze• facialexpression• locomotion• viseme

eyeseyes

BML = “complex/multimodal” BML ≠ “simple/unimodal”(that’s at the level of body)

torsotorso

legslegs

Coordination

89

What is BML? - ExampleWhat is BML? - Example <bml> <speech id=”s1”> <text>This is an <sync id=”wb3”> example</text> </speech>

<head id=”h1” type=”NOD” stroke=”s1:start”/>

<gesture id=”g1” stroke=”s1:wb3” relax=”s1:end” type=”BEAT”> <description level=”1” type=”MURML”>... </description> </gesture>

<gaze id=”z1” target=”PERSON1” stroke=”g1:stroke-0.1”/>

<posture id=”p1” shape=”RELAXED” start=”after(s1:end)”/>

<cadia:operate target=”SWITCH1” stroke=”p1:ready”/>

90

Work on BMLWork on BML

On-line wiki, publication at IVA07

Join the small and active community

IVA09 conference 14-16 Sept 2009 iva09.dfki.de

BML ws: 13 Sept 2009

91









92

Conducting with parametrised Conducting with parametrised kinematic hand motionkinematic hand motion

•H. van Welbergen, HMI, UT work in progress

93

Conducting – body motion with Conducting – body motion with physical modelphysical model


94

Fat/thin conductorFat/thin conductor


95

ConductorsConductors30 30 in in real-time real-time


96

Image morphingImage morphing- - automatic identification of feature pointsautomatic identification of feature points- interpolationinterpolation- per region (e.g. lip sync)per region (e.g. lip sync)- db/analysis of facesdb/analysis of faces- combination of morphing and 3d transformationscombination of morphing and 3d transformations


Face and gesture recognitionFace and gesture recognition

- - elicitation of characteristics for modeling (off-elicitation of characteristics for modeling (off-line)line)- real-time recognition of conveyed meaningreal-time recognition of conveyed meaning- user modeling for interaction control, on-lineuser modeling for interaction control, on-lineuser analysis to learn about how they react to user analysis to learn about how they react to VHsVHs- multimodal recognitionmultimodal recognition

97


Image analysis and synthesisImage analysis and synthesis-NPAR (e.g. line drawing) from photosNPAR (e.g. line drawing) from photos- exaggerationexaggeration- ageing, tiredness and other effectsageing, tiredness and other effects

Texturing Texturing - LOD- LOD

98

[email protected]

99

From: J. Walker, L. Sproull, R. Subramani: Using a Human Face in an Interface, CHI’94, pp 85-91.

Interacting with Virtual Humans Zsófia Ruttkay Associate Professor Human Media Interaction, Dept....

Documents

Transcript of Interacting with Virtual Humans Zsófia Ruttkay Associate Professor Human Media Interaction, Dept....