The JAST Collaborative Human-Robot Dialogue SystemMary Ellen Foster, Markus Rickert, Michael BraunRobotics and Embedded Systems Group, Department of Informatics, Technical University of Munichhttp://www6.informatik.tu-muenchen.de/

Overall goal of project● Build jointly-acting autonomous systems that collaborate and work

intelligently on mutual tasks in dynamic, unstructured environments.

Research areas● Cognitive and neural bases of joint action● Dialogue and joint action● Joint action in autonomous systemshttp://www.jast-net.gr/

Task: User and robot jointly construct Baufix models

... User You should

insert the red bolt in the central hole of this slat.

JAST Which red bolt? There are three red bolts available.

User Any red bolt. JAST [picks up

nearest red bolt] Here is a red bolt. Can you insert it while I hold the slat?

U+J [action] JAST We need a nut

to secure this bolt....

System architecture

EMET

Object referencing andscene overview

Feature linking

Short-termmemory

Long-termmemory

Object recognition

Speech processing

Syntax analysis

Dialoguemanagement

Recipeexecution

Face tracking

Gazetracking

Face positiontracking

Action primitivesequence building

Manipulators

Action primitives

Motion planning

Collision detection

Hand tracking

Pointing detection

Gesture detection

Feedback

Facialexpressions

Speech

Low-levelerror handling

Gaze control

Joint actioncoordination