JAST, 2020; 53:35-60 Submitted: 20.08.2019 Accepted: 04.03 ...
The JAST Collaborative Human-Robot Dialogue System · The JAST Collaborative Human-Robot Dialogue...
1
The JAST Collaborative Human-Robot Dialogue System Mary Ellen Foster, Markus Rickert, Michael Braun Robotics and Embedded Systems Group, Department of Informatics, Technical University of Munich http://www6.informatik.tu-muenchen.de/ Overall goal of project ● Build jointly-acting autonomous systems that collaborate and work intelligently on mutual tasks in dynamic, unstructured environments. Research areas ● Cognitive and neural bases of joint action ● Dialogue and joint action ● Joint action in autonomous systems http://www.jast-net.gr/ Task: User and robot jointly construct Baufix models ... User You should insert the red bolt in the central hole of this slat. JAST Which red bolt? There are three red bolts available. User Any red bolt. JAST [picks up nearest red bolt] Here is a red bolt. Can you insert it while I hold the slat? U+J [action] JAST We need a nut to secure this bolt. ... System architecture EMET Object referencing and scene overview Feature linking Short-term memory Long-term memory Object recognition Speech processing Syntax analysis Dialogue management Recipe execution Face tracking Gaze tracking Face position tracking Action primitive sequence building Manipulators Action primitives Motion planning Collision detection Hand tracking Pointing detection Gesture detection Feedback Facial expressions Speech Low-level error handling Gaze control Joint action coordination
Transcript of The JAST Collaborative Human-Robot Dialogue System · The JAST Collaborative Human-Robot Dialogue...
The JAST Collaborative Human-Robot Dialogue SystemMary Ellen Foster, Markus Rickert, Michael BraunRobotics and Embedded Systems Group, Department of Informatics, Technical University of Munichhttp://www6.informatik.tu-muenchen.de/
Overall goal of project● Build jointly-acting autonomous systems that collaborate and work
intelligently on mutual tasks in dynamic, unstructured environments.
Research areas● Cognitive and neural bases of joint action● Dialogue and joint action● Joint action in autonomous systemshttp://www.jast-net.gr/
Task: User and robot jointly construct Baufix models
... User You should
insert the red bolt in the central hole of this slat.
JAST Which red bolt? There are three red bolts available.
User Any red bolt. JAST [picks up
nearest red bolt] Here is a red bolt. Can you insert it while I hold the slat?
U+J [action] JAST We need a nut
to secure this bolt....
System architecture
EMET
Object referencing andscene overview
Feature linking
Short-termmemory
Long-termmemory
Object recognition
Speech processing
Syntax analysis
Dialoguemanagement
Recipeexecution
Face tracking
Gazetracking
Face positiontracking
Action primitivesequence building
Manipulators
Action primitives
Motion planning
Collision detection
Hand tracking
Pointing detection
Gesture detection
Feedback
Facialexpressions
Speech
Low-levelerror handling
Gaze control
Joint actioncoordination
