MIN FacultyDepartment of Informatics

Immersive Teleoperation and Simulation withROS & Unity3D

Current Status and Progress

Dennis Krupke

University of HamburgFaculty of Mathematics, Informatics and Natural SciencesDepartment of InformaticsTechnical Aspects of Multimodal Systems

29. November 2016

D. Krupke – ROS2Unity 1 / 13

IntroIntegrating ROS and Unity3D Applications Current and Future Work

D. Krupke – ROS2Unity 2 / 13

Table of ContentsIntegrating ROS and Unity3D Applications Current and Future Work

1. Integrating ROS and Unity3D2. Applications3. Current and Future Work

D. Krupke – ROS2Unity 3 / 13

Why?Integrating ROS and Unity3D Applications Current and Future Work

I usage of ROS is obvious for roboticsI Unity3D is state-of-the-art in interactive 3D computer graphics

and virtual reality (VR)

BenefitsI Fast prototyping of HRI scenarios with high quality graphical

representations and support of various input and outputdevices.

I Separating details of robot control in ROS from interactiondesign

D. Krupke – ROS2Unity 4 / 13

StructureIntegrating ROS and Unity3D Applications Current and Future Work

D. Krupke – ROS2Unity 5 / 13

Communication ExampleIntegrating ROS and Unity3D Applications Current and Future Work

D. Krupke – ROS2Unity 6 / 13

ROS-messages and JSONIntegrating ROS and Unity3D Applications Current and Future Work

I implemented messagesare (de)serializable toJSON strings

I serialized JSON stringsare transmitted

D. Krupke – ROS2Unity 7 / 13

Thanks to SebastianUseful packagesIntegrating ROS and Unity3D Applications Current and Future Work

Robot Importer (URDF) for UnityI parses URDF and creates hierarchical structure of the robot in

Unity3D using the 3D meshes from the ROS repositoriesI adds joints, which are controllable from scriptsI converts between different coordinate systems

Inverse Kinematics in Unity3DI bio-inspired inverse kinematics for 6-DOF posesI highly scalableI supports branching chainsI supports multi targets

⇒ In combination with the communication component viaROSbridge, Websockets and JSON it provides a powerful toolkit.D. Krupke – ROS2Unity 8 / 13

Excursion: Microsoft HoloLensIntegrating ROS and Unity3D Applications Current and Future Work

D. Krupke – ROS2Unity 9 / 13

SimulationIntegrating ROS and Unity3D Applications Current and Future Work

I training for non-experts in roboticsI safe testing of algorithmsI prototyping of HRI scenariosI psychological experiments

D. Krupke – ROS2Unity 10 / 13

TeleoperationIntegrating ROS and Unity3D Applications Current and Future Work

Topics of interest:I simulated latenciesI embodiment techniquesI methods for assigning different DOF of user and robot

D. Krupke – ROS2Unity 11 / 13

Example: Cartesian Path PlanningIntegrating ROS and Unity3D Applications Current and Future Work

D. Krupke – ROS2Unity 12 / 13

Ongoing WorkHigh-level VR-based teleoperationIntegrating ROS and Unity3D Applications Current and Future Work

Currently under development:Live-pointcloud view in 3D/VR.

Near future work:Integration of eye-tracking inside the HMD.

Final goal:Using eye- and hand-tracking for endpoint prediction ofreach-to-grasp gestures for reachable objects in a pick-and-placescenario.

D. Krupke – ROS2Unity 13 / 13