Proceedings of the 2006 IEEE/RSJInternational Conference on Intelligent Robots and Systems

October 9 - 15, 2006, Beijing, China

Design of a Control Strategy for Teleoperation of aPlatform with Significant Dynamics

Mattias Bratt, Christian Smith, and Henrik I. ChristensenCentre for Autonomous Systems

KTHStockholm, Sweden

Email: {bratt,ccs,hic}(kth.se

Abstract - A teleoperation system for controlling a robot withfast dynamics over the Internet has been constructed. It employsa predictive control structure with an accurate dynamic model ofthe robot to overcome problems caused by varying delays. Theoperator interface uses a stereo virtual reality display of therobot cell, and a haptic device for force feed-back includingvirtual obstacle avoidance forces.

Index Terms - Internet, prediction, remote bilateral tele-operation, time delay, haptics.

I. INTRODUCTION

Teleoperation is widely used in control of systems inmany different types of applications, such as robot assistedsurgery, space missions, handling of dangerous materials, etc.Most of these applications are characterized by limiteddynamics. An obvious question is thus "what are therequirements for design of systems in the presence ofsignificant dynamics?" A typical example of a task that isinherently dynamic is the catching of a flying object, e.g. aball.

As part of the Neurobotics project one of the topicalstudies is related to teleoperation for such scenarios. Auto-nomous catching of such objects ([1] and [2]) or hitting them[3] has been reported in the literature. For teleoperation thecontrol loop is closed through the operator. Here one of thekey questions is the use of models. Humans have a longexperience of catching flying objects and the performancedepends upon the information presented to the operator andhow this information matches to our models. Models for ballcatching by humans have been studied extensively as forexample reported by McIntyre [4].

To enable studies of use of such models for control, ahigh performance manipulation system (Fig. 1) has beendesigned for teleoperated ball catching. It was built from off-the-shelf components, and is further described in section III.

In this paper we describe the design of the teleoperationcontrol strategy for the control of the system. To demonstratethe generality of the presented approach, a related applicationof mobile platform teleoperation is also presented. Thedifferences between the two designs are described as well.

The research presented in this paper is funded in full by the 6th EUFramework Program, FP6-IST-00 1917, project name Neurobotics.

Fig. I The manipulator used for teleoperation studies

Initially the control strategy is discussed in Section II.The implementation of the chosen strategy is then presented inSection III - detailing both the catching and mobility systems.Early experimental results are presented in Section IV. Anumber of issues and an associated strategy for future studiesare presented in Section V. Finally the study is summarized inSection VI.

II. DESIGN OF A CONTROL STRATEGY

As mentioned earlier, teleoperation is widely used in alarge number of applications today. Direct teleoperation has anumber of different challenges depending on the problem athand. Some of the challenges include:0

0

0

Kinematic transferHandling of time delaysFeedback generationKinematic transfer refers to the fact that in some cases

there is a difference in kinematic structure between theactuation system (the manipulator) and the control systemavailable to the operator. An example is the transfer from aforce joystick to a manipulator. Another is the traditionalcommand console used for operating cranes on trucks.

Handling of time-delay is another challenge to ensurestability and maximum performance. It can be divided intohandling of deterministic time-delays and handling of

1-4244-0259-X/06/$20.00 C)2006 IEEE1700

stochastic time-delays. In control theory the handling ofdeterministic delays is often modeled using a Smith predictorstructure, whereas there are a number of different strategiesfor handling of stochastic delays such as [5].

Finally the issue of feedback generation is important asmultiple modalities often can complement each other andprovide efficient embedding of the operator in the scenario.

For the ball catching system and mobile platform we havechosen to use stereoscopic modeling and a force-reflectancejoystick to provide a maximum of flexibility in the feedbackgeneration and best performance in the type of experiments tobe performed. The study is performed in the context of controlof a system over the internet, so the system is assumed to havestochastic time characteristics.

A. Design ofan overall control structureA common technique for closing a teleoperation control

loop over a communication medium with significant delays isthe use of wave variables [6]. Whereas this can guaranteepassivity (no net output of energy) of the communication link,and thereby help prove stability of the complete system, itdoes impose severe performance penalties in a scenario wherefast dynamics are required despite the presence of substantialcommunication delays. For this reason a different approach,requiring an accurate dynamic model of the robot and itsenvironment, was selected. In the employed scheme, all forcesdisplayed to the operator through the haptic device aregenerated interacting exclusively with a dynamic simulationrunning on the user interface computer. Since this interactiondoes not involve a time delay, the risk of instabilities is muchreduced. The modeling of the robot is facilitated by the factthat the model includes the controller local to the robot.Knowing the input command and the dynamic state of therobot, the future state can be predicted well after a time of thesame magnitude as the communication delays.

Al. Basic structure ofthe teleoperation control systemTo close the control loop over the robot via the

communication link, we use a non-linear, multvariate Smithpredictor control structure (depicted for the robot armteleoperation case in Fig. 2). Because the system simulated(the robot and its controller at the robot site) in the predictor ishighly non-linear, it is not sufficient to correct only thepredictor output by the measured state y (the joint angles andspeeds) arriving from the robot, as in a standard Smithpredictor [7]. The simulation itself also needs access to thecorrected state estimate to be able to use the correct dynamicsfor that particular point in state space. The simulation outputhowever, is not corrected, and can still be compared toincoming measurements for generating new corrections.

The command and measurement communication delays,-c and -m respectively, are handledby

i) Adding an artificial delay la to the stochastic commanddelay -c when a command packet arrives at the robot sothat their sum, the virtual delay -c, is constant [8].

ii) Delaying the simulation result ysim(t+±v) by Tv+Tm before itis compared to the measured state y(t-cm) to form thesimulation correction 6. This means that when ameasurement packet arrives from the robot, the currentvalue of Tm is calculated from timestamps and an oldsimulation result ysim(t--m) retrieved from memory forcomparison.The net effect of this is that the simulation, the haptic

controller, and the operator all perceive time -c ahead of realtime. That allows the generated command signal u to travel tothe robot before it is needed by the robot controller. Iv valuesof up to 200 ms have been tested successfully.

Because the correction loop closed over thecommunication link runs at a much lower frequency ( 20 Hz)

Fig. 2 Robot arm control structure (skewed boxes represent delays)

1701

than the inner loop containing the simulation and control ofthe haptic device ( 500 Hz), the correction signal 6 must below pass filtered to reduce zero order hold noise that wouldotherwise be felt by the operator as vibrations. The low passfilter also reduces the high frequency gain of the correctionloop which moderates the spike effect of quick state changesin the simulation and at the robot being slightly offset in time.

A2. Systems with multiplefeedback channelsFor the mobile Pioneer II platform, feedback of

measurement data is slightly complicated by its being dividedinto two parts: relative odometry position measurements, andabsolute, but less frequent, laser scanner localization data. Theodometry data is calculated by the motion controller board ofthe Pioneer, and sent to the operator interface computer every50 ms. Laser localization is done less frequently based on datafrom a SICK scanner on the robot. The modified controlstructure appears in Fig. 3. The differences from Fig. 2 are thatthere are now two measurement delays, Tmo and -m,, forodometry and laser data respectively (Tmi>Tmo because of thetime required for computing the localization), and,correspondingly, two corrections, 6o and 61, which are bothadded to the simulation result Ysim to form the correctedestimate Ycorr of the robot state. 61 is formed as a correction ofthe odometry data yo, and is filtered through a slower low passfilter than 60 because of its lower update frequency.Furthermore, though not visible in the diagram, the correctionscan no longer be applied by plain vector addition because thestate y now contains a rotation (apart from the two planarCartesian coordinates, their first derivatives and the rotationalvelocity).B. Feedback generation

The basic principle of the haptic controller of Fig. 2 andFig. 3 is to map the haptic device handle to the chosen

command space (velocity or, in the case of teleoperating therobot arm, alternatively position) and virtually attach it by aspring to the current corrected simulation value of thecorresponding part of the robot state. I.e., when teleoperatingthe robot arm by velocity control, the current three-dimensional velocity of the end-effector is mapped to a pointin the haptic device workspace to which the handle is attractedby a virtual spring. The command sent to the robot is just theposition of the handle translated to a velocity by the inverse ofthe same mapping. In this way the force perceived by theoperator is related to the resistance of the robot and itscontroller to adjust to the command.

In addition, virtual viscous damping is applied to thehandle to help stabilize the system, and in the case of velocitycontrol, a weak force toward zero velocity is added as well. Toreflect the quality of the communication link, both of theseforces can be strengthened in response to any sustainedincrease in the measurement communication delays.

B]. Obstacle avoidanceforcesTo avoid known obstacles, these are mapped into the used

command space, and made to repel the handle with a forceproportional to the inverse of the square or cube of thedistance to the handle depending on the dimensionality of thespace. For position control this mapping is trivial, butmapping obstacles to velocity space requires some thought.The chosen mapping is based on the principle that a point invelocity space is part of an obstacle if, starting at this velocityv, and at the current position of the robot, braking with apostulated constant negative acceleration abrake to zero velocityresults in a collision. The distance traveled during such ahypothetical braking maneuver from speed v is

s

2v

.2abrake (1)

Fig. 3 Mobile robot control structure (skewed boxes represent delays)

1702

III. IMPLEMENTATION OF THE CONTROL MODEL

As mentioned earlier, teleoperation is studied in thecontext of the manipulation system shown in Fig. 1. It usesPowerCube actuator modules from Amtec, and is controlled ata frequency of 700 Hz by a standard PC using a CAN bus. Itskinematic configuration is of Puma560 type, allowing the useof analytical solutions for inverse kinematics and dynamics.Maximum end effector velocity and acceleration are 7 m/s and140 m/s2, respectively. The local controller PC communicateswith the user interface computer by UDP/IP.

In addition, mobile platform experiments are performedwith an ActivMedia Pioneer II robot with sonar and lasersensors. It has an onboard single board computer, whichhandles UDP/IP wireless communication with the userinterface computer, and uses ActivMedia's Aria API forinterfacing the motor controller board and sensors of the robot.

For interaction with the operator a stereoscopic displayand a force reflectance joystick are used. The latter is anOmega unit from Force Dimension, which provides a largeworkspace, as well as high stiffness and force output.

For the stereoscopic display two different alternativeshave been evaluated. The traditional stereoscopic display withshutter glasses has been tested using inexpensive glasses froman eDimensional 3D Vision System kit. However, Sharp hasrecently started to ship displays that allow free fusion ofstereoscopic information without glasses. We use a Sharp 3Dlaptop (model Actius RD3D). The basic visualization isperformed using Open Inventor 5 from Mercury ComputerSystems and the force reflectance is implemented using thelow level API provided by Force Dimension.

A. Design of VR environmentThe 3D graphics rendering is done using Open Inventor 5

from Mercury Computer Systems on top of OpenGL. Itemploys hierarchical scene graphs containing shapeprimitives, surface and lighting properties, andtransformations.

For building the scene graph a system of C++ classeshave been constructed, allowing high level specification ofrooms and their subcomponents, including windows, furniture,etc. The same C++ objects define the free space used inobstacle avoidance for the mobile platform.

The visual appearance of objects can be imported byOpen Inventor as VRML files and inserted into the scenegraph. This is very useful for complex objects, and makes itpossible to do the modeling of e.g. a chair in any 3D modelingpackage instead of having to define it in the C++ source code.A screen dump of the operator interface together with aphotograph of the same scene is in Fig. 4.

OpenGL quad buffered stereo with asymmetric frustumperspective mapping is supported, and the dependence ofteleoperation performance on any static or dynamic virtualcamera position can be explored. A web cam feed on aseparate connection acts as a reference.

B. Obstacle avoidanceforce generationAs mentioned in section II part B 1 above, force

generation is based on mapping obstacles to the commandspace used. To achieve this when commands are given asvelocities, obstacles are mapped to velocity space using therule that if the robot would collide with an obstacle even if itstarted braking immediately, then its current velocity is part ofan obstacle in velocity space.

B]. Mobile platform 2DforcesBecause the differential drive Pioneer II platform is non-

holonomic, it cannot be controlled in Cartesian velocity space,but is instead controlled in 2D speed-curvature (vK) space,with the haptic device locked to a plane. This is similar todriving a car, where the wheel controls the curvature, and theaccelerator and brake dictate the speed. A drawback is that,though physically possible on a differential drive platform, itdoes not allow turning on the spot as this would requireinfinite curvature. Additional problems result from not beingable to calculate the curvature from measured wheel speeds

rig. 4 visualization (lelt) ot tne robot in a room witn some lumiture (right).

1703

near zero velocity, but these can be handled by using anexponentially weighted (by traveled distance) moving averagefilter, in combination with a weak influence from the currentcommanded curvature.

Mapping of obstacles to VK space has been implementedusing sampling of VK space at 40x40 points evenly distributedaround the current (v,K) of the handle. In Fig. 5 the not yetsampled points are represented by dots, and the (v,K) of thehandle by a square. For each curvature, speeds successivelymore different from the current speed are sampled in thepositive and negative direction until a collision is found (across in the figure) or all points have been sampled (circles).

To build the obstacle avoidance force, each foundcollision point contributes a force proportional to the inverseof its squared distance to the handle and in the direction awayfrom the (v,K) of the collision point.

Points in VK space are sampled by calculating the positionin 2D Cartesian space where the robot would stop if brakingfrom speed v while following an arc with curvature K. Then allobstacles are checked to see if this position is occupied, whichwould cause a collision. To correct for the non-zero size of therobot, obstacles are expanded by the radius of the robotplatform (approximating it as cylindrical). A visualization ofthe positions corresponding to the sampling points is in Fig. 6.Dark points are on a constant curvature arc ending at acollision, light points are not.

The current implementation ignores the fact that thelength of the obstacle curve in VK space approximated by thesampled collision points is not proportional to the number ofpoints. Ideally, the force influence of line segments connectingneighboring collision points should be used instead.

The complete process of finding collision points andcalculating the avoidance force (involving sampling of 1600points if no collisions are found) is repeated at frequencyequal to that of the haptic controller ( 500 Hz).

B2. Robot arm 3DforcesIn the three dimensions of the Cartesian velocity space of

the robot arm, a similar sampling approach (involving 403sampling points) becomes infeasible. However, in this case apoint on an obstacle is easily mapped into velocity space sincethe distance to the point where the end-effector would stopafter constant deceleration is proportional to the square of theinitial velocity (Eq. 1), and the direction of course the same asthat of the velocity. This leads to the relationship

00000000

00000000

v

0 0 0 0 0 00 0 0 0 0 00 0 0 0 0 00 0 0 0 0 0

*3

0 0 0 0 0 0000000**0 0 0 0 0 00 00 00 0

000

X0000

V

* * x o x * a* x o o o x xx oo00 X XX

0 0 000 0

0 0 0 0 0 00000000o o o0000000o o o o o o _o

0 0 0 0 0 0 0

o o o o o o o

Fig. 5 Sampling of speed-curvature space, start and end of process.

rig. 6 voints sampleci ror oDstacles (oDstacle not stiown).

v =C r(r.r)'1/

(2)

where r is the position of a point on an obstacle relative tothe robot, v is the same obstacle point mapped into velocityspace, and C is a constant that depends on abrake. Using this, anobstacle surface can be mapped into velocity space, and theresulting obstacle avoidance force vector on the handle of thehaptic device becomes

F = ff Vh-V dQ.

Q((Vh V) .(Vh V))2(3)

Here, Vh is the velocity pointed to by the handle and Q isthe obstacle surface in velocity space.

Sampling an obstacle surface in Cartesian space,converting the samples to velocity space, and estimating thedQ surface element (whose analytical formulation is a bitcomplicated even for a plane in Cartesian space) using thedistance between converted samples allows estimation of theintegral. The spacing of the sampling points must probably beadjusted according to the variation of the force contribution(the integrand in (3)) to keep the required number of pointsdown. Obstacle avoidance forces in three dimensions have yetto be implemented.C. Internet communication delays

The communication medium between the operatorinterface and the robot is the Internet or another IP network.Whereas this gives several advantages, including world-wideavailability and abundant hardware and software support, italso poses some problems. There are no bandwidthguarantees, and, most importantly, there is a stochasticcommunication delay.

The magnitude and distribution of the delay [9] dependson several factors, such as physical distance, time of day etc.Typical round trip times are about 50 ms inside a city, 100 msUS coast to coast, and 150-200 ms for a transatlantic

1704

A

connection. Different kinds of fluctuations in the delay includerandom noise, changes due to variable network load, delaysfrom transmitting too much data, bunching of packets notoriginally transmitted together, and lost packets [9]. Improvedreal time communication properties can be achieved by usingIPv6 where available [10].

On top of the IP protocol, UDP (user datagram protocol)is used for the teleoperation communication. For transmissionof real time data UDP is preferable to TCP (transport controlprotocol), because it avoids the overhead of e.g. detecting andretransmitting lost packets, at the expense of not being able toguarantee transmission of all data or constant ordering [5]. Ifthe packets are self contained, i.e. they each contain samplesof all the transmitted signals, only the newest packet availableat the receiving side is relevant, making retransmission of olddata and enforcing constant ordering pointless.

To avoid the problems caused by variable delays, theteleoperation system uses a constant virtual delay equal to apostulated maximum communication delay as detailed insection II part Al above. The virtual delay is realized byembedding a timestamp in each packet and adding an artificialdelay at the receiving side so that the sum of thecommunication delay and the artificial delay is equal to thevirtual delay. It is chosen to make occasions when thecommunication delay is larger than the chosen value relativelyrare. They can then be treated as exceptions by the controlsystem causing the robot to stop gracefully.

IV. EARLY EXPERIMENTAL EVALUATION

Teleoperation using the proposed control structure hasbeen demonstrated for the mobile platform, and for the robotarm. In each case stable operation has been achieved forvarying delays up to 200 ms.

The manipulator system has only recently becomeavailable and early experiments with obstacle avoidanceforces were thus performed on the mobile platform. It hasbeen teleoperated in the presence of significant clutter tochallenge the operator and to require both visual and forcefeedback. The test environment used for early experiments isthat shown in Fig. 4 above.

With obstacles mapped to velocity space, their velocitydistance from the robot (and therefore their influence on theobstacle avoidance force) depends not only on the Cartesiandistance, but also on the speed of the robot. When driving fastthrough a narrow passage the generated force almostcompletely takes over control of the robot and guides it alonga good path. Conversely, when there is a lot of free space infront of the robot or if driving very slowly, almost no forcewill be generated.

V. ISSUES FOR FUTURE RESEARCH

The present system has been designed to provide aflexible basis for studying teleoperation of systems withsignificant dynamics, and for investigating several different

questions. The obvious questions to be studied include

* Design of visual feedback strategies to assist in grasping.* Types of force feedback to assist in grasping.* Balance between autonomous control and operator

interaction.* Impact of better communication performance (a la IPv6)

on teleoperation.* Use of human motion models for design of a model based

control strategy.* Selection of visual frame of reference to maximize human

performance.

These are merely a few of the questions to be studied inthe design of efficient interfaces.

VI. SUMMARY

The present paper has described the design of a systemfor teleoperation experiments in the presence of significantdynamics. The overall control model was developed and itwas demonstrated how the model can be used with one ormore sensory feedback modalities. In addition the basicfeedback models were presented. Initial experiments with amobile platform and with a manipulator system demonstratesthat the system has a high fidelity of reality in the presence ofstochastic delays of up to 200 ms. Finally a number of issuesfor future studies were outlined.

REFERENCES[1] Frese, U., et al., "Off-the-shelf vision for a robotic ball catcher," Proc.

IEEEIRSJ Int. Conf Intelligent Robots and Systems, Maui, Hawaii,USA, pp. 1623-1629, Oct. 2001.

[2] Borst, C., Fischer, M., Haidacher, S., Liu, H., and Hirzinger, G., "DLRHand II: Experiments and experiences with an anthropomorphic hand,"Proc. IEEE Int. Conf Robotics and Automation, Taipei, Taiwan, pp.702- 707, Sept. 2003.

[3] Senoo, T., Namiki, A., and Ishikawa, M. ,"High-speed batting using amulti-jointed manipulator," Proc. IEEE Int. Conf Robotics andAutomation, New Orleans, LA, pp. 1191-1196, April 2004

[4] McIntyre, J., Zago, M., Berthoz, A., and Lacquaniti, F., "Does the brainmodel Newton's laws?," Nature Neuroscience 4, pp. 693-694, 2001

[5] Munir, S., and Book, W.J., "Internet-based teleoperation using wavevariables with prediction," IEEEIASME Trans. Mechatronics, Vol. 7,no. 2, pp. 124-133, June 2002.

[6] Niemeyer, G. and Slotine, J.-J.E., "Using wave variables for systemanalysis and robot control," Proc. IEEE Int. Conf Robotics andAutomation, Albuquerque, NM, pp.1619-1625, April1997.

[7] Smith, 0. J. M., "A Controller to Overcome Dead Time," ISA Journal 6(2): 28-33, 1959

[8] Kosuge, K., Murayama, H., and Takeo, K., "Bilateral feedback controlof telemanipulators via computer network," Proc. IEEEIRSJ Int. ConfIntelligent Robots and Systems, Osaka, Japan, pp. 1380-1385, Nov.1996.

[9] Niemeyer, G., and Slotine, J.-J. E., "Toward bilateral Internetteleoperation," in Beyond webcams: an introduction to online robots,Goldberg, K. and Siegwart, R. Eds., ISBN: 0-262-07225-4, MIT Press,Cambridge, MA, 2001.

[10] Nufio, E., and Basafiez, L., "Using force feedback for assisted robotteleoperation via IPv6 protocol in high-tech networks," 36thInternational Symposium on Robotics (ISR), Tokyo, Japan, Nov. 2005.

1705