Physically Situated Language Interaction: an integrative ... · Attention models Engagement models...
Transcript of Physically Situated Language Interaction: an integrative ... · Attention models Engagement models...
![Page 1: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/1.jpg)
Dan Bohus
Physically Situated Language Interaction:
an integrative-AI challenge
![Page 3: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/3.jpg)
channel
signal
intention
conversation
understanding
planning
turn taking
engagement
communicative
competencies
![Page 4: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/4.jpg)
channel
signal
intention
conversation
understanding
planning
turn taking
engagement
communicative
competenciessituational context
who: physical awarenessidentify, track, and characterize relevant
actors, objects, states and relationships
what: situation and activitysense and reason about relevant events
and activities of self and others
why: goals and intentionssense and reason about beliefs,
intentions, goals and long-term plans
![Page 5: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/5.jpg)
situational context
who: physical awarenessidentify, track, and characterize relevant
actors, objects, states and relationships
what: situation and activitysense and reason about relevant events
and activities of self and others
why: goals and intentionssense and reason about beliefs,
intentions, goals and long-term plans
intention
conversation
understanding
planning
signalturn taking
channelengagement
communicative
competencies
![Page 6: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/6.jpg)
situational context
attentionintention
conversation
understanding
planning
signalturn taking
channelengagement
communicative
competencies
![Page 7: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/7.jpg)
Speaker:
Listener:
Speaker:
Listener:
Speaker:
Listener:
from Conducting Interaction: Achieving Mutual Orientation at Turn Beginning
![Page 8: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/8.jpg)
Speaker:
Listener:
Listener:
Speaker:
from Conducting Interaction: Achieving Mutual Orientation at Turn Beginning
![Page 9: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/9.jpg)
![Page 10: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/10.jpg)
![Page 11: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/11.jpg)
Infers attention on various targets
Relies on ML model for geometric visual attention
Leverage features from visual subsystems
Defined at phrase level
Specified at onset and production
Define expected targets
≠
… … … Excuse me! … … … To get … … … To get to 3800
Phrase 1
Wait Interject Wait Speak 2w Wait Speak phrase
Phrase 2
go to […]
![Page 12: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/12.jpg)
![Page 13: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/13.jpg)
![Page 14: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/14.jpg)
sensing and computation details
![Page 15: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/15.jpg)
![Page 16: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/16.jpg)
![Page 17: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/17.jpg)
![Page 18: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/18.jpg)
![Page 19: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/19.jpg)
system incorrectly infers user is not attending and inappropriately triggers pauses, interjections and restarts
![Page 20: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/20.jpg)
![Page 21: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/21.jpg)
situational context
intention
conversation
understanding
planning
signalturn taking
channelengagement
communicative
competencies
attention
![Page 22: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/22.jpg)
situational context
who: physical awarenessidentify, track, and characterize relevant
actors, objects, states and relationships
what: situation and activitysense and reason about relevant events
and activities of self and others
why: goals and intentionssense and reason about beliefs,
intentions, goals and long-term plans
intention
conversation
understanding
planning
signalturn taking
channelengagement
communicative
competencies
![Page 23: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/23.jpg)
![Page 24: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/24.jpg)
Microphone array capture
Sound source localization
Speech recognition
Language understanding
Infrared proximity sensors
Badge sensors
Face detection and tracking
Head-pose tracking
Facial feature tracking
Face identity recognition
Gender detection
Attention models
Engagement models
Turn-taking models
Behavioral control
Dialog management
Natural language generation
Speech synthesis
Avatar synthesis
Robot motion control
Floor-plan models
User models
complex (many components)programming models for coordinated computation; tools
![Page 25: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/25.jpg)
complex (many components)programming models for coordinated computation; tools
![Page 26: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/26.jpg)
Time and streams as 1st order citizens
double f; stream double f;
f=3; f=x*f-y;
Automatic persistence, historical access, sampling, transforms
act in real-time, under uncertaintyevolve programming languages? e.g. time & uncertainty
complex (many components)programming models for coordinated computation; tools
Uncertainty as 1st order citizen
double f; uncertain double f;
Representation, sampling, inference, belief updates
![Page 27: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/27.jpg)
Microphone array capture
Sound source localization
Speech recognition
Language understanding
Infrared proximity sensors
Badge sensors
Face detection and tracking
Head-pose tracking
Facial feature tracking
Face identity recognition
Gender detection
Attention models
Engagement models
Turn-taking models
Behavioral control
Dialog management
Natural language generation
Speech synthesis
Avatar synthesis
Robot motion control
Floor-plan models
User models
act in real-time, under uncertaintyevolve programming languages? e.g. time & uncertainty
complex (many components)programming models for coordinated computation; tools
integration of human- and machine-authored components
![Page 28: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/28.jpg)
Microphone array capture
Sound source localization
Speech recognition
Language understanding
Infrared proximity sensors
Badge sensors
Face detection and tracking
Head-pose tracking
Facial feature tracking
Face identity recognition
Gender detection
Attention models
Engagement models
Turn-taking models
Behavioral control
Dialog management
Natural language generation
Speech synthesis
Avatar synthesis
Robot motion control
Floor-plan models
User models
act in real-time, under uncertaintyevolve programming languages? e.g. time & uncertainty
complex (many components)programming models for coordinated computation; tools
integration of human- and machine-authored components
![Page 29: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/29.jpg)
act in real-time, under uncertaintyevolve programming languages? e.g. time & uncertainty
complex (many components)programming models for coordinated computation; tools
integration of human- and machine-authored componentsengineering of integrated learning systems
Engineering of integrated learning systems
learning in connected systems (new frontiers for ML & software engineering?)
learning in interactive settings: online, lifelong vs. batch
![Page 30: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/30.jpg)
act in real-time, under uncertaintyevolve programming languages? e.g. time & uncertainty
complex (many components)programming models for coordinated computation; tools
integration of human- and machine-authored componentsengineering of integrated learning systems
meta-reasoning & system-level (self)-optimization
Engineering of integrated learning systems
learning in connected systems (new frontiers for ML & software engineering?)
learning in interactive settings: online, lifelong vs. batch
Meta-reasoning and system-level (self)-optimization
self-monitoring and diagnosis / blame assignment
self-optimization
![Page 31: Physically Situated Language Interaction: an integrative ... · Attention models Engagement models Turn-taking models Behavioral control Dialog management Natural language generation](https://reader033.fdocuments.us/reader033/viewer/2022050514/5f9df2ec1f380278836edaee/html5/thumbnails/31.jpg)
act in real-time, under uncertaintyevolve programming languages? e.g. time & uncertainty
complex (many components)programming models for coordinated computation; tools
integration of human- and machine-authored componentsengineering of integrated learning systems
meta-reasoning & system-level (self)-optimization
machinelearning
softwareengineering
programminglanguages
distributedsystems
?