Teleoperation of Mobile Service Robots Over theNetwork: Challenges and Possible Solutions

Student: Stanko Kruzic, Advisor: Josip MusicUniversity of Split

Faculty of Electrical Engineering, Mechanical Engineering and Naval ArchitectureSplit, Croatia

{skruzic, jmusic}@fesb.hr

Abstract—With the ubiquity of service robots, various usagescenarios are developed for them, usually with a degree ofautonomy. However, due to dynamic nature of the environment,algorithms for autonomous operation might fail. Besides that,there are tasks which are not meant to be completed duringautonomous operation, and where the operator must take controlof the robot to complete them. In order to efficiently and safelyteleoperate the robot, the operator has to have a high degree ofsituational awareness. This can be achieved with the appropriatehuman-computer interface (HCI) so that the remote environmentmodel constructed with sensor data is presented at an appropriatetime and in an appropriate manner, and that robot commandscan be issued intuitively and without much effort. Networklatency problem is also addressed and a method for reducingits impact on teleoperation is presented.

I. PROBLEM FORMULATION

Service robots are made to assist humans by completingtasks that are repetitive, dirty or are done in hazardous environ-ments [1], [2] and are nowadays used for various applicationsin everyday life, both at home and in the industry. Someof them are designed so that they can perform their taskautonomously, while others usually have limited degree ofautonomy and require human intervention. Thus, occasionallyit is necessary that the operator takes control of the robot froma remote location. This is referred to as teleoperation.

Teleoperation requires communication between the robotand the remote operator, usually using 2.4 GHz WiFi or 3G/4Gmobile data network to connect to the Internet, so it can bereachable. There are situations when teleoperation is very diffi-cult or even impossible due to the presence of high or changingnetwork latency. This is because, in the presence of highlatency, teleoperation performance degrades, making feedbackof robot actions visible to the operator with a delay, whichmay result in a damage, both to the robot and the environment.Since all sensor data and robot commands are timestamped,the latency is measured as time difference between currentmoment and the moment of the timestamped data (either foruser-to-robot or robot-to-user case). For efficient teleoperationof the robot, the operator must feel immersed, i.e. must be wellaware of robot’s state and state of its environment. This is oftenreferred to as situational awareness, which is understanding ofthe conditions of dynamic environment and may be defined as”The perception of the elements in the environment within avolume of time and space, the comprehension of their meaning,

and the projection of their status in the near future.” [3].When teleoperating a robot, an operator has to be aware ofthe consequences of its actions both for the robot and forthe environment. This can be achieved with an appropriatehuman-computer interface (HCI). HCI generally consists oftwo main parts: 1) output device (a graphical interface) whichpresents all necessary feedback from the robot about its stateand the state of the environment, and 2) input device which isused to issue commands to the robot. The interface shouldbe designed in a way that it offloads as much workloadas possible from the operator and that makes robot controlintuitive and easy. Generally, more (sensor) data about robot’sstate and it’s environment improve situational awareness, butit has to be taken into account that too much informationmay distract the operator and overload network capacity. So,it’s a matter of optimisation to choose which informationshould be presented to the operator while not congesting thecommunication channel.

The manner in which information from sensors is presentedto the operator also affects the performance of teleoperation[4]. Graphical interfaces containing multiple information fusedtogether in a single augmented reality (AR) frame, were shownto be more efficient than arrangement when they were pre-sented in multiple frames, side-by-side. We even demonstratedin [5] that for simple tasks there is no statistically significantdifference between visual and teleoperation scenarios whenappropriate data is presented in the interface (see Fig. 1). Itshould be noted that inclusion of HCI control components thatcontain virtual reality elements [6], or haptic-based elements[7] might further improve the teleoperation experience.

II. PROBLEM SOLUTION

We propose a two-pronged approach to the problem. Firstwe chose the best HCI feedback and control elements so toachieve optimal use of available information channel capacity.Initial research has been carried on the topic in [5]. Gamepadand gaming steering wheel were used as input devices whilegraphical interfaces with and without depth information whichmade a total of 4 different HCI setups. In a user study,operators had to complete three steering tasks, and resultsshowed no significant difference between used HCI setups andalso no significant difference between teleoperation and visualcontrol. However, it was noted that teleoperation parameters

Fig. 1. AR graphical interface with depth information (green arrows) overlaidover live camera image and with 2D map of the environment, used in [5]

degraded somewhat, comparing to those of visual control.This can be further augmented with the inclusion of humanin the autonomy control loop by means of explicit trajectorydefinition. Our proposal is to use an approach based on ARwith a touch-panel of mobile phone or tablet (see Fig. 2),similar to the one proposed in [8]. This may be seen asa combination of teleoperation and autonomous navigation,since the operator is presented with feedback data on thescreen, like in teleoperation, but does not steer the robot itself,it rather provides an input (goals or a whole trajectory) for thenavigation which robot completes autonomously.

Fig. 2. A sketch of the proposed input interface

The second solution in the two-pronged approach addresseslatency issues. We recognise that latency is difficult to controland/or minimise in public general purpose networks. Thus, wepropose one of the possible solutions to the latency problem:in the case of high (or highly variable) latency robot switchesto autonomous control seamlessly, and returns control to theoperator when conditions are appropriate for efficient and safenavigation. This approach is depicted in Fig. 3. The switchingthreshold will be defined empirically and based on availableliterature like [9]. For this approach to be possible, the finalgoal needs to be known in advance. Also, some kind ofhuman control model needs to be available. This approachcan also be used to aid the operator in teleoperation taskswhen long/tidies navigation from point A to point B is needed.The manner in which autonomous operation is to be executedis currently being researched. The method being developedis inspired by one described in [10], adapted for wheeledmobile robots. It features a self-supervised learning method.Data is collected in simulation by generating simple straight

LatencyMeasurement

DNNRobot

Human

Switchinglogic

Sensors

goal + pose−

Fig. 3. Block diagram of the proposed solution

trajectories and treat ones that resulted in crashing a robotinto walls as negative samples and ones that did not resultin crashing as positive samples. The samples are then usedto train deep neural network (DNN) that may act as robotcontroller in autonomous operation later, which will generateappropriate velocity commands that will steer the robot to thedesired location, avoiding obstacles in the way. The idea lookspromising (in initial testing) because this method does notrequire a map or any other knowledge about the environmentthe robot will operate in. Algorithm for human/autonomousswitch also needs to be developed.

III. CONCLUSIONS

The paper describes mobile robot teleoperation and its pre-requisites and presents parts of results of initial research thatwas conducted in order to find the optimal HCI configurationfor teleoperation. The autonomous navigation system withthe innovative approach based on self-supervised learning ispresented, and ideas how to combine it with teleoperation aregiven. Some preliminary testing on it was done, and the idealooks promising.

REFERENCES

[1] A. Birk, K. Pathak, S. Schwertfeger, and W. Chonnaparamutt, “Theiub rugbot: an intelligent, rugged mobile robot for search and rescueoperations,” in IEEE International Workshop on Safety, Security, andRescue Robotics (SSRR). IEEE Press, vol. 10, 2006.

[2] V. Celan, I. Stancic, and J. Music, “Cleaning up smartcities—localization of semi-autonomous floor scrubber,” in InternationalMultidisciplinary Conference on Computer and Energy Science(SpliTech). IEEE, 2016, pp. 1–6.

[3] M. R. Endsley, “Design and evaluation for situation awareness enhance-ment,” in Proceedings of the human factors and ergonomics societyannual meeting, vol. 32, no. 2. SAGE Publications, 1988, pp. 97–101.

[4] C. W. Nielsen and M. A. Goodrich, “Comparing the usefulness of videoand map information in navigation tasks,” in Proceedings of the 1st ACMSIGCHI/SIGART conference on Human-robot interaction. ACM, 2006,pp. 95–101.

[5] S. Kruzic, J. Music, and I. Stancic, “Influence of human-computerinterface elements on performance of teleoperated mobile robot,” inMipro 2017, 40th Jubilee International Conference on Information andCommunication Technology, Electronics and Microelectronics, 2017.

[6] J. Jankowski and A. Grabowski, “Usability evaluation of vr interface formobile robot teleoperation,” International Journal of Human-ComputerInteraction, vol. 31, no. 12, pp. 882–889, 2015.

[7] W. Li, Z. Liu, H. Gao, X. Zhang, and M. Tavakoli, “Stable kinematicteleoperation of wheeled mobile robots with slippage using time-domainpassivity control,” Mechatronics, vol. 39, pp. 196–203, 2016.

[8] S. M. Jung, D. Choi, K. Kwon, and J. W. Jeon, “A sketch-based interfacefor remote robot control on an online video screen,” in ConsumerElectronics (ICCE), 2013 IEEE International Conference on. IEEE,2013, pp. 428–429.

[9] S. Vozar and D. M. Tilbury, “Driver modeling for teleoperation withtime delay,” IFAC Proceedings Volumes, vol. 47, no. 3, pp. 3551–3556,2014.

[10] D. Gandhi, L. Pinto, and A. Gupta, “Learning to fly by crashing,” arXivpreprint arXiv:1704.05588, 2017.