Prosodic Entrainment Across Cultures - Columbia Universityjulia/talks/toronto17.pdfProsodic...

ProsodicEntrainmentAcrossCultures

SecondIntonationWorkshopUniversityofTorontoFebruary17,2017

JuliaHirschberg,ColumbiaUniversity

6/25/17 2

Collaborators

• Rivka Levitan Columbia University• Zhihua Xia,Tongji University• Agustín Gravano, University of Buenos Aires• Štefan Beňuš, Constantine the Philosopher

University

6/25/17 3

The Chameleon Effect

• Entrainment/Alignment/Adaptation– “In conversation, people tend to adapt their

communicative behavior to that of their conversational partner.” (Giles et al ’87)• Non-consciousmimicry ofthepostures,mannerisms,facialexpressions,andotherbehaviorsofone'sinteractionpartners(Chartrand &Bargh 1999).• Perception-behaviorlink:theunderlyingmechanismfortheChameleonEffect--- Unintentional,non-consciouseffects ofsocialperceptiononsocialbehavior"(Chartrand,Maddux,&Lakin,2005)

EntrainmentinMultipleDimensions

• Lexical and syntactic (Brennan ‘00, Reitter et al ‘07)• Acoustic/Prosodic (Matarazzo et al ‘68, Jaffe & Feldstein

‘70, Natale ‘77, Cappella & Planalp ‘81, Street ‘84, Sherlom & La Riviere ‘87, Guitar & Marchinkoski ‘01)

• Phonological/Phonetic (Pardo ‘06)• Socio-cultural (Azuma ‘97, Roth ‘05)• Jokes and laughter (Bales ‘50, Raganath et al ‘11)• Facial expression and gesture (Mauer & Tindall ‘83,

Hale & Burgoon ‘84, Chartrand & Bargh ‘99)• Posture (Condon & Ogston ‘67)

5

Why is Entrainment Important?

• Subjects who entrain – Perceived as more socially attractive (Putnam & Street '84,

Bourhis et al '75)– Perceived as more competent (Street '84)– Conversation perceived as more intimate (Buller & Aune

'88)• Entrainment leads subjects to like their conversational partners

(and their computers) more and to perceive interactions as more successful (Nass et al ‘95, Chartrand & Bargh ‘99)

• Long-term syntactic entrainment is a good predictor of actual task success in Map Task (Reitter et al ’07)

TheColumbiaGamesCorpus(Gravano)

• 12spontaneoustask-orienteddyadicconversations(9h8mspeech)

• 2subjectsplayseriesofcomputergames,noeyecontact(45m39smeansessiontime)– 2sessionspersubject,w/differentpartners

• Multiplegamesandtypes• Recordedonseparatechannelsinsoundproofbooth,digitizedanddown-sampledto16k

• FeaturesextractedwithPraat

6/25/17 6

6/25/17 7

TheCardsGame

6/25/17 8

TheObjectsGame

Describer: Follower:

Tongji GamesCorpus(Xia)

• 115spontaneoustask-orientedsessions– 70pairsofspeakers(40female,30male)– 12hoursofrecordeddialogue

• UniversitystudentswithaNationalMandarinTestCertificatelevel2,gradeAorabove

• Elicitedusingtwogames:PictureOrdering(roleimbalance),PictureClassifying(cooperative)

9

A B

PictureOrderingGame

A B

PictureClassifyingGame

Unitsofanalysis

• Inter-pausal unit(IPU): Pause-freesegmentofspeech(50msormore)fromasinglespeaker

• speech<silence>speech<silence>speech• Turn:Sequenceofspeechfromonespeakerwithoutinterveningspeechfromtheotherspeaker.

• Session:Completeinteractionbetweentwosubjectsononetask

12

Unitsofanalysis

• Inter-pausal unit(IPU): Pause-freesegmentofspeech(50msormore)fromasinglespeaker

• Turn:Sequenceofspeechfromonespeakerwithoutinterveningspeechfromtheotherspeaker.

• Session:Completeinteractionbetweentwosubjectsononetask

13

IPU IPU IPU

LowLevelProsodicFeatures

14

• Intensitymean• Intensitymax• Intensitymin• F0mean

• F0max• F0min• speakingrate

FormsofEntrainment(Levitan &Hirschberg‘11)

Proximity ---- significantsimilarityofpartnerfeatures

Convergence ----significantincreaseinsimilarityofpartnerfeaturesovertime

Synchrony ---- correlatedrelativechangeinpartnerfeatures

Similarity/Proximity

• Globalorlocal?• Exactorrelative?• Convergentorconstant?

16

Synchrony


17

Convergence


18

DefiningGlobalPairwiseEntrainment• Similarity

• Synchrony:positivecorrelationbetweenpartners• Convergence:negativecorrelationbetweenpartners

Baselinesimilarity

Partner similarity

Selfsimilarity

LocalEntrainment:Proximity

20

IPU

ti

IPU

ti+1

IPU

tj

IPU

tk

IPU

tl

LocalEntrainment

21

IPU

ti

IPU

ti+1

IPU

ti+2

IPU

ti+3

IPU

ti+4

IPU

ti+1

IPU

ti+2

IPU

ti+3

Results (Levitan ‘14, Xia et al ‘14)

22

Global similarity

Local similarity

Synchrony Globalconvergence

Local convergence

Feature SAE MC SAE MC SAE MC SAE MC SAE MCIntensity mean ✓✓ ✓ ✓ ✓ 0.35 0.63 0.08Intensity max ✓(✓) ✓ ✓ ✓ 0.33 0.55 0.08Pitch mean 0.07 0.66 0.08 0.22Pitch max ✓ 0.04 0.61 ✓ 0.10 0.24Jitter − − 0.15 − − −Shimmer − − 0.07 − − 0.04 −NHR − ✓ − 0.12 − ✓ − 0.03 −Speaking rate ✓ ✓ ✓ ✓ 0.13

ComparingMandarinandStandardAmericanEnglish

• Similarity– Globalandlocal:similarintensity,rate

• Synchrony– StrongersynchronyforMC(intensity,pitch)

• Convergence– GlobalforSAEonly(pitch,NHR,rate)– StrongerlocalconvergenceforMC(pitch)

23

TheoriesofEntrainmentandGender

• Dominance–MaleDominanceHypothesis– CommunicationAccommodationTheory

• Perception– Perception-BehaviorLink– Femaleperceptualsensitivity

24

PredictionsandPreviousWork

– Predictions• InMFconversations,femalesshouldentrainmore.• ThereshouldbemoreentrainmentinFFconversationsthanMMconversations.

– Previouswork• Bilous andKrauss(1988)• Namy etal.(2002)• Pardo (2006)

25

Partnervs.Non-PartnerDifferences

26

ENTp = − speaker1 f − speaker2 f

ENTx = −speaker1 f −Xi, fi∑

X

X = set of speakers of the same gender and role as speaker1’s partner who are never paired

with speaker1.

EntrainmentandGender

27

Feature FF MM FM

MC SAE MC SAE MC SAE

Intensity mean ✓ ✓ ✓ ✓ ✓

Intensity max ✓ ✓ ✓ ✓ ✓

Intensity min -- -- --

F0 mean ✓ ✓

F0 max ✓ ✓

F0 min -- -- --

Speaking rate ✓ ✓ ✓ ✓ ✓ ✓


28

Feature FF MM FM





F0 mean ✓ ✓

F0 max ✓ ✓

F0 min -- -- --



29

Feature FF MM FM





F0 mean ✓ ✓

F0 max ✓ ✓

F0 min -- -- --



30

Feature FF MM FM





F0 mean ✓ ✓

F0 max ✓ ✓

F0 min -- -- --


Conclusions

• MM<<FF<<MF• SimilarpatternsforMandarinChineseandSAE

31

SocialDimensionsofEntrainment(Levitan etal‘12)

• Recall that subjects who entrain are– Perceived as more socially attractive (Putnam & Street '84,

Bourhis et al '75)– Perceived as more competent (Street '84)– Speech perceived as more intimate (Buller & Aune '88)

• Entrainment leads subjects to like their conversational partners (and their computers) more and to perceive interactions as more successful (Nass et al ‘95, Chartrand & Bargh ‘99)

• Long-term syntactic entrainment a good predictor of actual task success in Map Task (Reitter et al ’07)

6/25/17 32

AnnotationofSocialVariables

• AmazonMechanicalTurkworkerslabeled168ColumbiaGamesCorpusobjectgames

• Answeredfollowingquestionsaboutpartners– Doess/hebelieves/heisbetterthanhis/herpartner?– Makingitdifficultforhis/herpartnertospeak?– Seemengaged inthegame?– Seemtodislike his/herpartner?– Iss/hebored withthegame?– Directing theconversation?– Frustrated withhis/herpartner?– Encouraging his/herpartner?

6/25/17 33

– Tryingtodominate theconversation?– Makinghim/herselfclear?– Planning whats/heisgoingtosay?– Polite?– Tryingtobeliked

• Questionsabouttheconversation– Doesitflownaturally orisitawkward?– Aretheparticipantshavingtroubleunderstandingeachother?

– Whichpersondoyoulikemore?– Whowouldyouratherhaveasapartner?

6/25/17 34

Hypotheses

• CommunicationAccommodationTheory– Givingencouragementpositivelycorrelatedwithentrainment

– Conversationalawkwardness negatively• Similarity-AttractionTheory– Tryingtobelikedshouldbepositivelycorrelated

• DependencyOver-Accommodationoccurswhenaninterlocutorappearsdependentonaspeaker,givingtheimpressionthatthespeakeriscontrollingtheconversation(West&Turner,2009).

6/25/17 35

Findings

• BasedonCommunicationAccommodationTheory– Givingencouragementpositivelycorrelatedwithentrainment

– Conversationalawkwardness(weakpositivecorrelation)

• BasedonSimilarity-AttractionTheory– TryingtobelikedpositivelycorrelatedNocorrelationbetweenperceiveddominanceandentrainment

6/25/17 36

EntrainmentinHigherLevelProsodicFeatures(Gravano etal‘14)

• ToBI annotationofColumbiaGamesCorpus– ThreeexpertlabelersusingtheToBI conventions:– Tonaltier:targetsintheF0contour• Pitchaccents:H*,L*,L+H*,L*+H,H*+L,downstep• Phraseaccents:H-,L-,!H-• Boundarytones:H%,L%.

– Orthographictier:time-alignedwords– Breakindextier:degreesofjuncture(0-4)–Misc tier:laughs,disfluencies,etc.

EntrainmentonPitchContoursandSocialVariables(Gravano etal’14,‘15)

• Measuresofcontoursimilaritybetweenspeakers– PerplexityoflanguagemodelsofsequencestrainedonSpeakerAandusedtomodelprosodicsequencesofSpeakerB:lowperplexityindicatesgreatersimilarity

– Levenshtein distanceofsimilarintonational phrasecontoursusedbySpeakerAandSpeakerB:lowvaluesshowsimilarcontoursareutteredclosertogether

– Kullback-Leibler divergencebetweencontoursofSpeakerAandSpeakerB:lowvaluesshowthatoneisasubsetoftheother

• Howsimilar areSpeakerA’scontoursareSpeakerB’s?

ε1Measure:N-GramPerplexity

• HowwelldoesaprosodicmodeltrainedonApredictB’sprosody?– SpkrA:L*L-H%H*H-L%...– SpkrB:H*!H*H- L*H-H%...– TRAINTRIGRAM–MODELTESTε1(A,B)=−(PerplexityofA'smodelonB'sproductions)

– Lowerperplexitymeansmoresimilar

ε2Measure:Levenshtein Distance

HowfarawayfromB’sproductionofcontourcisA’sproductionofthesamecontour?

ε3Measure:Kullback-Leibler Divergence

Experiments

• Builda24Dvectorwiththevalueofεi foreachmemberofeachspeakerpair

• Buildsimilarvectorforeachsocialvariablev(e.g.,bored-with-game) whereAj,Bj arethetwospeakersfromsessionj

• RunPearson'scorrelationtestsbetweenentrainmentvectorsandsocialvariablevectors

CorrelationswithSocialVariables

SocialVariable Perplexity Levenshtein Dist KLDivergenceMaking-self-clear pos pos

Givingencouragement pos pos

Engaged-in-game neg pos pos

Contributes-to-successful-task-completion pos pos pos

Trying-to-be-liked pos

Planning-what-to-say pos pos

Dislikes-partner neg

Making-it-difficult-for-partner-to-speak pos pos

Bored-with-game neg neg neg

ConclusionsandFutureResearch

• 3novelmetricsofentrainmentonintonational contoursannotatedwithintheToBI framework.

• Findings:correlationsofprosodicentrainmentwithperceivedlevelsof– speakerengagement– positivepartner-orientedfeaturesofsocialbehavior(givingencouragement,makingselfclear,etc.)

• Futurework:Automatecomputationofourmeasuresusingautomaticprosodiclabelingtools(e.g.,AuToBI).

EntrainingtoUsersinSpokenDialogueSystems(Levitan etal‘16)

46

GoFish:DoUsersPreferanEntrainingSystem?

47

GoFishHelpers

Method

• 19participants:– 9female,10male– Ages20—35

• Eachsession:~45userturns(entraining+control)– ~9minutes– Acoustic-prosodicfeaturesextractedbyPraat– Advicelogged

50

UserPreferencesforEntrainingHelpers

• Trust– “Whogavebetteradvice?”N.S.– Implicittrust(whoseadvicefollowed?)Entraining

• Liking– “Whichadvisordidyoulikebetter?”Entraining

• Voice– “Whosevoicedidyoulikebetter?”Entraining– “Strange” Non-Entraining– “Annoying” Non-Entraining

51

EntrainmentinPorteño SpanishandSlovakGoFish-with-HelpersGames

• DifferencesfromEnglish:– Systementrainedonlyonspeechrate– notintensity

– DifferentTTSsystems,avatargender(female),variedpitchrange

– Noeffectforentrainingavatar• Why?– IssueswithidentifyingspeechrateaccuratelyfromASR…

CurrentandFutureResearch

• Entrainmentandtrust:GoFish,NavGame (HarryPotterlikeadventuregame),GuessWho (akaTwentyQuestions)gamesbeingdevelopedandtestedforSlovakandSpanish

• EnglishFishingExpeditionusingcrowd-sourcing– Subjectsrankconversationswithdifferentformsofentrainmenttoseewhichcombinationsbestpredictsubjectchoices

• Entrainmentindeceptivespeech

Thanks!

• This work has been supported by – NSF IIS-0803148 – Humanities and Social Sciences Foundation of

China Education Ministry (11YJA740113) – Air Force Office of Scientific Research, Air Force

Material Command, USAF under Award No. FA9550-15-1-0055

– UBACYT 20020120200025BA

Publications

• Implementingacoustic-prosodicentrainmentinaconversationalavatar.R.Levitan,S.Benus,R.H.Galvez,A.Gravano,F.Savoretti,M.Trnka,A.WeiseandJ.Hirschberg.Interspeech 2016.

• EntrainmentandTurn-TakinginHuman-HumanDialogue.R.Levitan,S.Benus,A.Gravano andJ.Hirschberg.AAAISpringSymposiumonTurn-TakingandCoordinationinHuman-MachineInteraction2015.

• Acoustic-ProsodicEntrainmentinSlovak,Spanish,EnglishandChinese:ACross-LinguisticComparison.R.Levitan,S.Benus,A.Gravano andJ.Hirschberg.SIGDIAL2015.

• BackwardmimicryandforwardinfluenceinprosodiccontourchoiceinStandardAmericanEnglish,A.Gravano,S.Benus,R.Levitan,J.Hirschberg.Interspeech 2015.

• ProsodicentrainmentinMandarinandEnglish:across-linguisticcomparison.Z.Xia,R.Levitan,J.Hirschberg.SpeechProsody2014.(BestStudentPaperAward)

• ThreeToBI-basedmeasuresofprosodicentrainmentandtheircorrelationswithspeakerengagement.A.Gravano,S.Benus,R.Levitan,J.Hirschberg,SLT2014.

• Acoustic-ProsodicEntrainmentinHuman-HumanandHuman-ComputerDialogue,R.Levitan,PhDThesis,ColumbiaUniversity,2014.

55

• EntrainmentinSlovakCollaborativeDialogues,S.Benus,R.Levitan,J.Hirschberg,A.Gravano,S.Darjaa.5thIEEEInternationalConferenceonCognitiveInfocommunications,2014.

• Entrainment,DominanceandAllianceinSupremeCourtHearings.S.Benus,A.Gravano,R.Levitan,S.I.Levitan,L.Willson,J.Hirschberg.JournalofKnowledge-BasedSystems2014.

• Acoustic-prosodicentrainmentandsocialbehavior.R.Levitan,A.Gravano,L.Willson,S.Benus,J.Hirschberg,A.Nenkova.NAACL2012.

• Entrainmentinspontaneousspeech:ThecaseoffilledpausesinSupremeCourthearings.S.Benus,R.Levitan,J.Hirschberg.CogInfoCom 2012.

• Measuringacoustic-prosodicentrainmentwithrespecttomultiplelevelsanddimensions.R.Levitan,J.Hirschberg.Interspeech 2011.

• Entrainmentinspeechprecedingbackchannels.R.Levitan,A.Gravano,J.Hirschberg.ACL2011.

• Pitchsimilarityinthevicinityofbackchannels.M.Heldner,J.Edlund,andJ.Hirschberg.Interspeech 2010.

• Highfrequencywordentrainmentinspokendialogue.A.Nenkova,A.Gravano,J.Hirschberg.ACL/HTL2008.

Prosodic Entrainment Across Cultures - Columbia Universityjulia/talks/toronto17.pdfProsodic...

Documents

Transcript of Prosodic Entrainment Across Cultures - Columbia Universityjulia/talks/toronto17.pdfProsodic...