Post on 15-Jan-2016
Alignment in human-human and human-computer
interactionsHolly Branigan
Jamie Pearson
University of Edinburgh
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Collaborators
University of Edinburgh
• Martin Pickering
Stanford University
• Clifford Nass
• John Hu
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Outline• Our approach
• Features of Human-Human interaction (HHI)– Audience design
– Alignment• Patterns of alignment
• Explanations for alignment
• Relevance to HCI
• Methodology for studying HCI
• Data from experiments investigating lexical/syntactic alignment in HCI
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Our approach
• Psycholinguistic accounts of HHI:– Characterising linguistic behaviour
– Identifying underlying psychological mechanisms• Focus on lexical and syntactic processing
• Applying to HCI:– Predicting human (user) behaviour
– Simulating human behaviour
– Modifying human behaviour
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Features of HHI
• Linguistic behaviour in HHI is highly flexible and adaptive:– An individual speaker’s behaviour is partially
contextually-determined.
– Linguistic choices (semantic, lexical, syntactic, phonological..) choices may vary according to context.
– Relevance to HCI:• Features of HCI communicative context may influence
linguistic behaviour.
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Determinants of Behaviour
• Influence of addressee on speaker:• Indirectly – speaker’s beliefs about
addressee’s state of knowledge, interests etc– Audience Design
• Directly – via addressee’s own linguistic behaviour
– Alignment
• Both influences may be active simultaneously
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Audience Design• Tailoring utterance to fit addressee:
– Their knowledge, interests, beliefs etc
• May be based on speaker’s a priori beliefs or assumptions about addressee
• Or may be based on direct evidence from addressee– E.g., addressee’s feedback
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Audience Design – lexical choice• Fussell & Krauss (1992):
– speakers adjust their choice of lexical terms to fit the assumed knowledge of their addressees:
• Proper names when addressee likely to know referent:
Clint Eastwood
• More detailed descriptions when addressee less likely to know referent:
Media mogul, was married to Jane Fonda, owns CNN, has a moustache and grey hair
– Assessment of addressee knowledge is based on a priori judgements about social distribution of knowledge
• movie stars > industrialists
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Audience Design – lexical choice• Isaacs & Clark (1987):
– A priori assessment of addressee knowledge can be rapidly adjusted on basis of direct evidence from feedback
• Lexical choices then reflect this adjusted assessment
– ‘experts’ in a domain (New York landmarks) quickly adjust their choice of lexical terms to fit the apparent expertise of their addressees:
– Proper names when addressee is apparently fellow expert
Chrysler building
– More detailed descriptions when addressee is apparently non-expert
Building with a tall pointy roof and a spike on top
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Audience Design - syntax• Audience design may influence syntactic choices:
– Speakers may choose syntactic structures that are most easily understood by addressees.
• Ambiguity-avoidance:
Put the penguin in the cup on the star vs
Put the penguin that’s in the cup on the star
When two penguins (and an empty cup), temporary ambiguity at penguin in the cup…
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Audience Design - syntax• Audience design may influence syntactic choices:
– Speakers may choose syntactic structures that are most easily understood by addressees.
• Ambiguity-avoidance:
Put the penguin in the cup on the star vs
Put the penguin that’s in the cup on the star
When two penguins (and an empty cup), temporary ambiguity at penguin in the cup…
Speakers include that’s more often (removing ambiguity) when there are two
penguins than where is only one penguin.
Haywood, Pickering & Branigan (2005)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Audience Design - summary• Speakers in HHI may engage in strategic planning
of aspects of utterances:
– Calculations of addressee knowledge, beliefs etc
– Based on a priori assumptions or direct evidence
• A priori assumptions are quickly updated in light of direct evidence; audience design reflects this
– Strategic planning may affect lexical choice, syntax, semantic choices
• E.g., choice of reference frame (Schober, 1993)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment• Speakers are affected directly by addressee’s
linguistic behaviour:– Tendency for speakers in a dialogue to converge linguistic
behaviour.
– After hearing an interlocutor use particular linguistic behaviour, speakers tend to repeat that behaviour.
– This tendency towards alignment is robust and highly pervasive.
• Rhetorical structure, semantic structure, syntactic structure, lexical choice, accent, speech rate, ….
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment• Alignment can be implicit:
– Almost always arises without explicit negotiation• Speakers often align on different expression than negotiated
expression
– Speakers usually unaware of effects• Can sometimes (but often cannot) report awareness of
meaning-related alignment
• Usually unaware of alignment of form
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – semantic choices• Alignment occurs for aspects of language
associated with meaning:
– e.g. reference frames (Watson, Pickering & Branigan, 2005)
The dot is left of the camera vs The dot is below the camera
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – semantic choices• Alignment occurs for aspects of language
associated with meaning:
– e.g. reference frames (Watson, Pickering & Branigan, 2005)
The dot is left of the camera vs The dot is below the camera
– e.g., description schemas (Garrod & Anderson, 1987)
I’m at B5 vs I’m two along and two up
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – lexical choice• Also alignment of lexical choice:
– Use same words in same ways
– box vs node
– square = single node vs configuration of nodes
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – lexical choice• May align on unusual/rare lexical choice:
– e.g., if first person to name object is a non-native speaker
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – lexical choice• May align on unusual/rare lexical choice:
– e.g., if first person to name object is a non-native speaker
Rocking chair
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – lexical choice• May align on unusual/rare lexical choice:
– e.g., if first person to name object is a non-native speaker
Rocking chair
The chair that can go back and forth
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – lexical choice• May align on unusual/rare lexical choice:
– e.g., if first person to name object is a non-native speaker
Rocking chair
The chair that can go back and forth
You can shake your body
Bortfeld & Brennan, 1997
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment• Alignment at semantic and lexical levels may be
linked to different types of meaning:– Alignment of perspectives on a situation (ways of thinking
about the world)
– E.g., rainbow trout vs coloured fish
• But other alignment seems to be unrelated to convergence on types of meaning– e.g., speech rate
– e.g, syntax (alternatives express same meaning)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – syntactic choices
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – syntactic choices
The artist is selling the dancer the gun (Double Object)
vs
The artist is selling the gun to the dancer (Prepositional Object)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment – syntactic choices
The artist is selling the dancer the gun (Double Object)
vs
The artist is selling the gun to the dancer (Prepositional Object)
A: The chef handing the book to the teacher
B: The artist selling the gun to the dancer
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Branigan et al. (2000)• Picture-description/-matching game
– Stooge and naïve subject alternately describe pictures
– Stooge produces scripted descriptions (50% PO, 50% DO)
• Stooge describes source picture to subject:
– The nun showing the monk the banana
• Subject chooses matching picture:
– Yup
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Branigan et al. (2000)• Picture-description/-matching game
– Stooge and naïve subject alternately describe pictures– Stooge produces scripted descriptions (50% PO, 50% DO)
• Stooge describes source picture to subject: – The nun showing the monk the banana
• Subject chooses matching picture:– Yup
• Subject describes target picture to matcher:– The artist selling the dancer the gun
• Matcher chooses matching picture:– Uh huh
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Branigan et al. (2000)• Picture-description/-matching game
– Stooge and naïve subject alternately describe pictures– Stooge produces scripted descriptions (50% PO, 50% DO)
• Stooge describes source picture to subject: – The nun showing the monk the banana
• Subject chooses matching picture:– Yup
• Subject describes target picture to matcher:– The artist selling the dancer the gun
• Matcher chooses matching picture:– Uh huh
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Confederate-scripting paradigm
Confederate’s script
Naïve participant
Picture cards to be matched
Confederate
Bla
h
Bla
h
Bla
h
Branigan et al (2000)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment - syntax• Strong tendency to align syntax with interlocutor:
– More likely to produce a Prep Obj description after hearing a Prep Obj description than after a Double Obj description, and vice versa.
– 77% aligned descriptions when verb repeated
– 63% aligned descriptions when verb not repeated• Chance = 50%
– NB: speakers align sentence form: both alternatives have same denotational meaning
• (cf. semantic/lexical alignment)
– Speakers align dynamically (produce both PO and DO)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment - syntax• Similar effects found:
– for other structures:• E.g., NP structure: a red cat vs a cat that’s red (Cleland &
Pickering, 2003)
– NB: Relative clause = strongly dispreferred in null context
– in multi-party dialogues:• Speakers syntactically align, whether directly addressed or
not.
A: The waitress is giving the monk the banana
C: Yup
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment - syntax• Similar effects found:
– for other structures:• E.g., NP structure: a red cat vs a cat that’s red (Cleland &
Pickering, 2003)– NB: Relative clause = strongly dispreferred in null context
– in multi-party dialogues:• Speakers syntactically align, whether directly addressed or
not.
A: The waitress is giving the monk the bananaC: Yup
B: The teacher is handing the cowboy the jugC: Uh huh
Branigan et al. (in press)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment - syntax• Similar effects found:
– in different communicative contexts:• e.g., during walkie-talkie communication
– in ‘special’ populations such as bilinguals, L2 learners, children…
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment - syntax• Similar effects found:
– in different communicative contexts:• e.g., during walkie-talkie communication
– in ‘special’ populations such as bilinguals, L2 learners, children…
El cerdo llora El pingüino baila (SV)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment - syntax• Similar effects found:
– in different communicative contexts:• e.g., during walkie-talkie communication
– in ‘special’ populations such as bilinguals, L2 learners, children…
El cerdo llora El pingüino baila (SV)
Llora el cerdo Baila el pingüino (VS)
Flett, Branigan & Pickering, in prep
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment - summary• Alignment with addressee’s linguistic behaviour:
– is robust
– can be very strong determinant of behaviour
– occurs for many (all?) levels of linguistic structure
– occurs for aspects of structure concerned with form, not just meaning
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment and audience design• Alignment and audience design may co-occur.
Put the penguin in the cup… vs the penguin that’s in the cup…
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment and audience design• Alignment and audience design may co-occur.
Put the penguin in the cup… vs that’s in the cup…
– Priming: • tendency to say The penguin that’s in the cup after hearing
The sheep that’s on the plate
– Audience design:• tendency to say The penguin that’s in the cup when there are two
penguins Haywood, Pickering & Branigan, 2005
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Explanations of alignment• Social psychological approaches to alignment:
– Identification with a particular social group (inc. addressee)
– Reciprocity effects (linked to politeness norms?)
– Such approaches explain why alignment of linguistic form occurs (in absence of meaning differences)
– Perceived social identity of addressee is important• e.g., politeness is only relevant for addressees that are
perceived as social agents
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Explanations of alignment• Alignment as audience design:
– Alignment may reflect strategic effects (choosing to adopt other person’s perspective to enhance communication)
– Choosing e.g., the same description schema or referential expression maximises the chances of effective communication
– Such approaches do not really explain why alignment of linguistic form occurs (in absence of meaning differences)
– Perceived identity of addressee is important• e.g., depends on addressee being intentional agent
• inanimate entities do not have perspectives or beliefs
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Explanations of alignment• Alignment as an automatic behaviour:
– Primitive, default behaviour
– children:
• align significantly more than adults (e.g., NP structure: 75% vs 30%)
• tend to align form, even when this leads to misunderstanding: – e.g., using same term with different reference (e.g., square)
• must learn to suppress tendency towards alignment where appropriate (Garrod & Clark, 1994).
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Explanations of alignment• Alignment may be based on automatic priming
mechanisms:• activation-based account (Branigan et al., 2000)
– comprehension of word or structure activates associated linguistic representations
» VP V NP PP
– representations retain residual activation
– residual activation facilitates subsequent selection
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Explanations of alignment• Alignment may be fundamental to efficient communication
(Pickering & Garrod, 2004)
• Efficient communication arises when interlocutors come to have the same understanding of relevant aspects of the world.
– This arises from alignment of their situation models (e.g., Zwaan & Radvansky, 1998).
– Alignment of situation models arises from alignment of other aspects of language (e.g., syntax, lexical choice).
• Alignment at one level promotes alignment at others
Lexical alignment syntactic alignment semantic alignment
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Explanations of alignment• Explanations are not mutually exclusive: multiple
factors may underline alignment.
– Most likely that at least some implicit element• Participants generally report lack of awareness of alignment
– But other factors may also contribute to overall effect:• Basic (automatic) alignment effect may be enhanced by e.g.
social factors (e.g., social status)
• Alignment at some levels of structure may be differentially susceptible to additional influences
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Implications for HCI• Does alignment occur in HCI?
– Pickering & Garrod (2004):• alignment is fundamental to effective communication
– So evidence that alignment is absent in HCI would imply that HCI is necessarily ineffective/inefficient.
• May highlight inherent difficulties in HCI
– Conversely, evidence that alignment is present in HCI suggests possibility of effective communication in HCI
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Alignment in HCI?• Why might we expect alignment in HCI?
– Alignment seems to be a default linguistic behaviour:• May be consequence of architecture of human language
processor
• Might therefore expect to find it in any communicative context
– Existing evidence that people treat computers as social actors Reeves & Nass (1996)
• They interact with them and evaluate them as they do other people
People should align with computers
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Leveraging alignment in HCI• Alignment may be useful behaviour:
– Facilitates predictions of user behaviour• Reduces decision space in e.g., speech recognition
– Allows simulation of human behaviour
• Users may feel more comfortable with systems that display human-like behaviour
• Users might feel more positively towards systems that align than those that do not
– May allow [implicit] modification of human behaviour• Users may modify their linguistic behaviour to align with
system• May allow modelling of desired input
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Patterns of alignment in HCI• How might alignment in HCI pattern?
• If alignment is purely automatic priming:– Alignment would occur whenever a linguistic structure is
encountered
• But in HHI, speakers’ (lexical and syntactic) choices are also influenced by:
– a priori beliefs about addressee knowledge, capability etc
– direct evidence about addressee knowledge, capability etc
• Hence beliefs about the knowledge, capability etc of a system might influence the extent to which speakers align with it.
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Relevant factors in HCI alignment• A priori beliefs about social identity of system:
– If systems are treated as social agents just like humans, then alignment should occur just as with humans.
• But which group of humans would count as comparable social agents?
– Limited capability?
– If systems are not treated as social agents just like to humans, then alignment may differ:
• Less alignment if alignment has substantial component related to e.g. politeness
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Relevant factors in HCI alignment• Beliefs about system capability:
– Users may assume computers to be less capable (generally and/or specifically linguistically).
• May increase likelihood of aligning, relative to HHI – Possibly different degrees of alignment with different systems
• May increase likelihood of aligning with unusual structure/expression
– Effects of feedback:• A priori beliefs may be updated in light of feedback
• Positive feedback of understanding may lead to comparable alignment as in HHI
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Relevant factors in HCI alignment• Beliefs about addressee identity:
– Modality may affect beliefs about addressee identity:
• Use of synthesised speech may emphasise non-humanness of addressee
– Might enhance difference between HCI/HHI
• Use of (canned) speech may de-emphasise non-humanness of addressee
– Might reduce differences between HCI/HHI
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Summary• Alignment is potentially highly important in HCI –
but there are many factors that might affect patterns of behaviour.– Specifically, many reasons why alignment in HCI might
differ from alignment in HHI
• One important issue: – extent to which any differences between HCI and HHI are
artifact of communicative situation rather than genuine differences between HCI and HHI
• Involvement of computer in communication
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Methodology
Another person or a computer... or whatever
Identical in all conditions
?
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Methodology• Picture-description experiments (as in HHI):
– Single object: lexical choices– Actions: syntactic choices
• Manipulate participants’ beliefs about identity of addressee:
– Human (at other end of remote connection)
– Computer (at other end of remote connection)
• In fact, there is no interlocutor:
– Participants always interact with a computer program that produces pre-scripted utterances
– ‘Reverse Wizard-of-Oz’
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Methodology• Systematically control addressees’ utterances that
participant encounters:– Actual linguistic behaviour of addressee is always identical
in all conditions
• How do beliefs about addressee affect participants’ likelihood of lexical/syntactic alignment?– Differences in participants’ linguistic behaviour must be
due to differences in beliefs about interlocutor.
– NB: This method is also informative about alignment in computer-mediated HHI.
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Outline• Series of experiments investigating:
– Do people align with computers?
– Focus on syntactic and lexical alignment
• Basic effects
• Influences of e.g., modality, a priori beliefs, feedback
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Possible patterns• No alignment in HCI:
– Only align with interlocutors that can experience positive affect or have beliefs/knowledge
• Equal alignment in HCI and HHI:– Compatible with cognitive economy accounts
• More alignment in HCI than HHI:– Beliefs about interlocutor’s affect alignment e.g.
knowledge/capability
• (No alignment in HHI– People do not align in CMI)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic• Interlocutor
– Human vs. Computer
• Two separate experiments: Verb in interlocutor’s description– Same vs. Different in participant’s picture
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic• Interlocutor’s description
– Same PO: The pirate handing the cake to the sailor
– Same DO: The pirate handing the sailor the cake
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic• Interlocutor’s description
– Same PO: The pirate handing the cake to the sailor
– Same DO: The pirate handing the sailor the cake
– Different PO: The pirate giving the cake to the sailor
– Different DO: The pirate giving the sailor the cake
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic• Interlocutor’s description
– Same PO: The pirate handing the cake to the sailor
– Same DO: The pirate handing the sailor the cake
– Different PO: The pirate giving the cake to the sailor
– Different DO: The pirate giving the sailor the cake
• Participant’s picture
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic alignment paradigm
Interlocutor describes
PO or DO
Participant matches
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic alignment paradigm
Participant describes
PO or DO?
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic alignment paradigm
Participant describes
PO or DO?
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic alignment paradigm
Interlocutor matches
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results• Align with people and with computers
– No difference in the magnitude of alignment
0
0.2
0.4
0.6
0.8
1
computer person
Pro
port
ion o
f alig
ned r
esp
onse
s
Different Verb
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
0
0.2
0.4
0.6
0.8
1
Results• Align with people and with computers
• More alignment with computers than people
computer person
Pro
port
ion o
f alig
ned r
esp
onse
s
Same Verb
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Interim summary• Syntactic alignment occurs in both CMI and HCI
– Tendency to use same syntax as ‘human’ or ‘computer’ interlocutor
• When verb is repeated, more alignment in HCI than CMI– Only difference in in beliefs about interlocutor
– Interlocutor’s behaviour identical
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Interim summary• Beliefs about interlocutor can affect strategic
alignment– Overlies ‘basic’ (implicit, non-strategic) alignment
– Syntactic choices not usually open to introspection, hence to strategic influences – only obvious in same-verb condition
• Lexical choices are usually more open to introspection– Hence should be generally more susceptible to strategic
component
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical• Interlocutor
– Human vs. Computer
• Term used by the interlocutor– Preferred vs. Dispreferred term
• Experimental items pretested– Preferred term used more than 90% (e.g. bench)
– Dispreferred term acceptable (e.g. seat)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical alignment paradigm
Interlocutor describes
Preferred or dispreferred
term
Participant matches
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical alignment paradigm
Participant describes
Preferred or dispreferred
term?
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical alignment paradigm
Participant describes
Preferred or dispreferred
term?
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical alignment paradigm
Interlocutor matches
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results• Align with people and with computers
• More alignment with computers than people
computer person
0
0.2
0.4
0.6
0.8
1
Pro
port
ion o
f alig
ned r
esp
onse
s
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Interim summary• Lexical alignment occurs in both CMI and HCI
• Greater alignment with computers than people when the availability of such a strategy is obvious
• Beliefs about a system influence the extent that speakers align:– Strongly affects behaviour – >80% use of term that is
strongly dispreferred (<10% use in null context)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Interim summary• Differences in alignment due to differences in a
priori beliefs about interlocutor’s knowledge etc– Interlocutors’ linguistic behaviour identical in all conditions
• Alignment does not reflect beliefs based on feedback from computer’s earlier contributions– Received feedback from computer of understanding of
syntax and lexical choices
• Conservative approach to HCI: A priori assumptions not updated in light of direct evidence– Cf. HHI evidence
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic: Modality• Modality may affect salience of beliefs about
interlocutor identity– Use of synthesised speech may emphasise non-humanness
• Hence promote alignment
– Use of speech may deemphasise non-humanness• Hence reduce alignment
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Syntactic: Synthesized Speech• Syntactic alignment: PO/DO
• Interlocutor– Human vs. Computer
• Modality– Text vs. [Synthesized] Speech
• Interlocutor-text / Participant-text
• Interlocutor-text / Participant-speech
• Interlocutor-speech / Participant-text
• Interlocutor-speech / Participant-speech
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results• More alignment with computers than people
• No effect of modality
0
0.2
0.4
0.6
0.8
1
computer person
Pro
port
ion o
f alig
ned r
esp
onse
s
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical: Canned speech• Same factors as previous
– Spoken version
• Interlocutor’s “voice” pretested– Rated progressively distorted/impoverished versions for
plausibility as a computer or person
• Chosen voice equally plausible as a computer or a person
• Voice gender-matched with participant
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results
computer person
Pro
port
ion o
f alig
ned r
esp
onse
s
0
0.2
0.4
0.6
0.8
1
• Align with people and with computers
• More alignment with computers than people
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Combined analyses
0
0.2
0.4
0.6
0.8
1
Pro
port
ion o
f alig
ned r
esp
onse
s
0
0.2
0.4
0.6
0.8
1
Interlocutor-text / Participant-text Interlocutor-spoken / Participant-spoken
computer person
• Although in right direction, no significant effect of modality
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Interim summary• Increased alignment in HCI is robust effect
– Not reliably influenced by modality, whether synthesised or canned speech
• Beliefs about interlocutor that influence alignment seem to be impervious to relatively superficial and overt manipulations of ‘humanness’– Not a dimension that needs to be worried about in systems
design?
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical: Persistence of alignment• Alignment may persist over longer intervening
material if interlocutor believed less capable– More effort to encode and recall interlocutors’ choices in
order to maximise chances of effective communication
• If interlocutor believed to be capable:– Alignment with dispreferred term may occur in short-term
(politeness)
– But not over longer distances
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical: Persistence of alignment• Interlocutor
– Human vs. Computer
• Distance between interlocutor naming picture and participant subsequently naming it– Near vs. Far
• Near: Immediately preceding turn
• Far: Nine intervening filler turns
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results• Align with people and with computers
• More alignment with computers than people
0
0.2
0.4
0.6
0.8
1
Pro
port
ion o
f alig
ned r
esp
onse
s
computer near person nearperson farcomputer far
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results• Alignment with people decreases with increased
distance
0
0.2
0.4
0.6
0.8
1
Pro
port
ion o
f alig
ned r
esp
onse
s
computer near person nearperson farcomputer far
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results• Alignment with computers is maintained despite
increased distance
0
0.2
0.4
0.6
0.8
1
Pro
port
ion o
f alig
ned r
esp
onse
s
computer near person nearperson farcomputer far
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Interim summary• Need for reciprocity/politeness to other people
diminishes
• Maintain high level of alignment with computers– Conservative approach in HCI: Go to a great deal of effort
to maximise the chances of effective communication
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical: Capability• Alignment to computers:
– Robust effect
– Determined by a priori beliefs about addressee knowledge, capability etc
– Not influenced by direct evidence about addressee knowledge, capability etc
• Can we manipulate participant’s beliefs about knowledge, capability etc of interlocutor?
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical: Capability of computer• HCI condition only
• Interlocutor– ‘Basic’ computer vs. ‘Advanced’ computer
• Manipulate beliefs about capability– Actual behaviour identical in both conditions
– Only difference: start-up screen for 10 seconds prior to the experiment
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Basic computer manipulation
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Advanced computer manipulation
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results• Align with basic and advanced computers
• More alignment with basic than advanced computer
basic computer advanced computer
0
0.2
0.4
0.6
0.8
1
Pro
port
ion o
f alig
ned r
esp
onse
s
Pearson et al (CHI-2006)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Interim summary• Beliefs about computer’s capabilities influence
alignment
• Beliefs can be manipulated through simple, superficial, non-functional aspects of system– Not affected by actual behaviour of system (evidence of
understanding)
• Does conservatism wrt updating beliefs on basis of feedback in HCI reflect something specific to HCI?– Are speakers similarly conservative in HHI when
interacting with humans of unknown capability?
– Test group: Non-native speakers
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical: Capability of person• CMI conditions only
• Interlocutor– Native person vs. Non-native person
• Manipulate beliefs about capability– Actual behaviour identical in both conditions
– Only difference: participants told they were interacting with native or non-native person
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results• No difference in alignment with non-native than
native people
0
0.2
0.4
0.6
0.8
1
native person non-native person
Pro
port
ion o
f alig
ned r
esp
onse
s
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Interim summary• Greater alignment with basic than advanced
computer– Determined by a priori beliefs about interlocutor’s
knowledge, capability etc
– Not influenced by direct evidence about interlocutor’s knowledge/capability
• No greater alignment with non-native than native person– Influenced by direct evidence about interlocutor’s
knowledge/capability
• What if we remove direct evidence about person interlocutor’s knowledge/capability?
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Lexical: Capability of person(2)• Same factors as previous
• Interlocutor– Native person vs. Non-native person
• No feedback from interlocutor when matching– No direct evidence about interlocutor’s understanding
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Feedback
Interlocutor matches
Participant has
described
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
No feedback
Interlocutor matches
Participant has
described
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Results
0
0.2
0.4
0.6
0.8
1
native person non-native person
Pro
port
ion o
f alig
ned r
esp
onse
s
• Greater alignment with non-native than native people
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Beliefs about mental states• Interlocutors establish a “common ground” of
shared background knowledge Clark & Marshall (1981)
– Each interlocutor works out what they assume the other knows about the situation
• Common ground can be accrued on the basis of:– Linguistic co-presence: What has been said previously
– Community membership: What you can infer on the basis of them belonging to a particular social group
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Beliefs about mental states• Greater alignment with computers than people and
with basic than advanced computers– Computers have radically different mental states than us
– Unclear what capabilities computers have, so align to maximise the chances of effective communication
• Greater alignment with non-native than native speakers only in absence of feedback– Non-natives have similar mental states to us
• With feedback: beliefs are up-dated. Based on their contributions, non-natives are capable enough
• Without feedback: cannot assume they are capable, so align to maximise the chances of effective communication
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Conclusions• HCI is like HHI in important ways:
– Tendency to align with interlocutor
– Both lexical and syntactic alignment
• But important differences:– Differences in a priori beliefs about human vs computer
interlocutors influences magnitude of alignment
– Where different choices salient, speakers align more with computers than with people
• Strategic component additional to automatic component of alignment
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Conclusions• Beliefs in HCI can be manipulated easily
– Superficial non-functional aspects of system
• But beliefs in HCI are not readily updated on basis of behavioural evidence
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Interface design implications• Users adapt their behavior to fit the system. This
adaptation reflects beliefs about the system’s capability, and not its actual behaviour
• Improved natural language recognition– To elicit high degrees of alignment, a system should not
claim “humanness”• A computer that tries to act smarter than it actually is is
disliked Wang et al (2005)
– Consistent use of words/syntax by interface will implicitly limit users’ input, reducing burden on recognition systems
– May also have affective implications
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Future directions: Align to listener• Alignment/adaptation to one’s interlocutor
increases:– Efficiency executing conversational tasks Clark & Wilkes-
Gibbs (1986)
– Likelihood of achieving higher level goals such as changing beliefs Carenini & Moore (2000)
– Waitress’ tips van Baaren et al (2003)
– Social Psychology: Increased liking; marking as in-group …
• Are people affected by computers aligning with them?
• Do people prefer computers that align with them?
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Future directions: Align to listener • Alignment to the user by the system might affect
– Level of alignment by user (can be leveraged to improve recognition accuracy)
– Liking of the system and the other users
– Perceived credibility (expertise and trustworthiness)
– Purchase behaviour
– Task performance
– Web stickiness (amount of time spent on a site)
– Branding effectiveness (multiple visits to a site)
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Align to listener: Design• Interlocutor
– Human vs. Computer
• Interlocutor’s responses– Align vs. Misalign with participant’s
• Measure characteristics of participant’s responses– Align or misalign with interlocutor?
• Affective measures– Perceptions of the interaction, self, and the interlocutor
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Align to listener: Design
• Interlocutor: “boat”
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Align to listener: Design
• Interlocutor: “boat”
• Participant: “jacket”
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Align to listener: Design
• Interlocutor: “boat”
• Participant: “jacket”
• Interlocutor: “jacket” “coat”
align misalign
Holly Branigan & Jamie PearsonHow People Talk To Computers Workshop Hansewissenschaftskolleg 22nd April 2006
Align to listener: Design
• Interlocutor: “boat”
• Participant: “jacket”
• Interlocutor: “jacket” “coat”
• Participant: align and say “boat”?
align misalign