Learning Motion Skills from Expert Demonstrations andOwn Experience using Gaussian Process RegressionKathrin Gräve, Jörg Stückler, and Sven BehnkeAutonomous Intelligent Systems Group, Institute of Computer Science VI, University of Bonn, Germany

AbstractWhile today’s industrial robots achieve high performance in a wide range of tasks, the programming and adaptation ofsuch robots to new tasks is still time-consuming and far from convenient. This shortcoming is a major obstacle for thedeployment of robots in small to medium-size companies and also for service applications. Robot learning by imitationhas been identified as an important paradigm to tackle this practicability issue. However, such approaches are onlyapplicable in controlled static scenarios. To perform less constrained tasks, the robot needs the ability to generalize itsskills and to adapt to changing conditions. It is thus a natural idea to formulate the problem as learning from experienceand to incorporate demonstrations by an expert in the learning process. We present a new framework for learning ofmotion skills that seamlessly integrates demonstration learning from an expert teacher and further optimization of thedemonstrated skill by own experience. Our method employs Gaussian Process regression to generalize a measure ofperformance to new skills that are close to already learned skills in the state-action space. We evaluate our approach forthe task of grasping an object.

1 Introduction

Today’s industrial mass production would not be possiblewithout the invention of robots that efficiently carry outrepetitive manufacturing tasks. These robots usually workin an isolated, static environment and are programmed tofulfil a single, specific task. In the future, robots may beemployed to relieve humans of even more tasks which arecumbersome, monotone or even dangerous for humans inindustrial, as well as service applications. These robotswill have to be flexible enough to easily adapt to new tasksand unexpected changes in the environment.As it is not feasible to preprogram the robot for every sit-uation it may ever encounter, the development of intuitiveways to teach a robot is of central importance in this areaof research. For personal robots, this also requires intuitiveinterfaces that are accessible to inexperienced users and al-low owners of robots to adapt them to their personal needs.In order to achieve this, we combine imitation and rein-forcement learning in a single coherent framework. Bothways of learning are well known from human teaching psy-chology. Applying them to robot learning allows a teacherto intuitively train a robot.In our approach, the actions generated by either learningmodule are executed by the robot and their performance ismeasured as a scalar reward. We assume that the reward iscontinuous over the combined space of states and actionsand apply Gaussian Process regression to approximate itsvalue over the entire space. This allows to generalize expe-riences from known situations. Furthermore, it provides ameasure of uncertainty regarding the achievable reward in

any situation. In each situation, we decide for imitation orreinforcement learning based on a measure that trades offlarge reward and predictability of actions.For imitation learning, we extract the parameters of a con-troller from the demonstrated trajectory. This controller isable to generate anthropomorphic arm movements from asmall set of parameters. For reinforcement learning, wedetermine new actions that promise high reward but alsooffer a predictable outcome.The advantages of our approach are manifold: The way inwhich we integrate both learning paradigms allows each ofthem to mitigate the other’s shortcomings and improves theoverall learning quality. The imitaition learning methodnarrows the search space of reinforcement learning whichgreatly improves learning speed. Furthermore, it allows toacquire anthropomorphic movements from human demon-strations. This offers a huge advantange over manual pro-gramming as this kind of movements is usually complexand it is very hard to describe what makes a movementhuman-like. Using reinforcement learning helps to reducethe number of demonstrations that are required to success-fully learn a task. Furthermore, it is used to improve the re-sults of imitation learning and to compensate for the kine-matic differences between the human and the robot. Ineffect, our proposed learning method makes robot trainingaccessible to non-experts in robotics or programming.

2 Related WorkImitation learning methods comprise a set of supervisedlearning algorithms in which the teacher does not give the

ISR / ROBOTIK 2010

212

Figure 1: The teacher demonstrates a grasping movement (top). From the recorded trajectory, our approach extracts aparameterized motion primitive. The robot imitates the grasp by executing the extracted motion (bottom).

final solution to a problem but rather demonstrates the nec-essary steps to achieve that solution. Although imitationlearning has been studied for decades, its mechanisms arenot yet fully understood and there are many open questionsthat have been grouped into the broad categories relating tothe questions of whom to imitate, what to imitate, how toimitate and when to imitate [1].

Consequently, there are many approaches to apply imita-tion learning to robots [1]. In early approaches to program-ming by demonstration [2], motion is recorded by teach-inthrough guiding or teleoperation by a human expert. Thedemonstrated trajectories are played back to generate thedesired motion. Recently, Calinon proposed a probabilisticframework to teach robots simple manipulation tasks [3].Demonstrations were given by teleoperating a robot. Itsmotions in relation to the objects in the world were subse-quently encoded in a Gaussian Mixture Model after reduc-ing their dimensionality. By applying Gaussian MixtureRegression and optimization, the robot was able to repro-duce the demonstrated movements in perturbed situations.In order to facilitate this generalization, several demonstra-tions of a movement need to be given. From these demon-strations the robot captures the essence of the task in termsof correlations between objects.

Reinforcement learning offers a way to reduce the num-ber of required training examples. However, it is known tobe prone to the curse of dimensionality. Early approacheswere thus limited to low-dimensional and discrete state andaction spaces [4]. More recently, strategies have been pro-posed that allow for continuous representations. For in-stance, so called policy gradient methods have been devel-oped and successfully applied to optimize the gait of SonyAIBO robots [5]. These methods, however, discard most ofthe information contained in the training examples. To im-prove data-efficiency, Lizotte et al. [6] proposed to selectactions probabilistically, based on Gaussian Process Re-gression and the most probable improvement criterion. Wepropose an improved search strategy that not only makes

efficient use of available data, but also balances the proba-bilities for improvement and degradation.To overcome the limitations of the approaches above,Schaal [7] was among the first who proposed to combinereinforcement learning and imitation learning. In his work,a robot was able to learn the classical task of pole balanc-ing from a 30s demonstration in a single trial. In morerecent work, Billard and Guenter [8] extended their imita-tion learning framework by a reinforcement learning mod-ule, in order to be able to handle unexpected changes inthe environment. However, both approaches merely usedthe human demonstrations to initialize the reinforcementlearning, thus reducing the size of the search space. Inour approach, further demonstrations can be incorporatedat any point in time.

3 Expected Deviation in GaussianProcesses

Central to our approach is the idea that the performance ofmovements can be measured as scalar reward and that thismeasure can be generalized across situations and actions.Thus, we form a combined state-action space and define ascalar continuous value function Q on it.

3.1 Gaussian Process Regression

We apply Gaussian Process Regression (GPR, [9]) to gen-eralize value across the state-action space and to cope withthe uncertainty involved in measuring reward and execut-ing actions. The basic assumption underlying GaussianProcesses (GPs) is that for any finite set of points X ={xi}Ni=1 the function values f(X) are jointly normal dis-tributed. In GPR, observations yi at points xi are drawnfrom the noisy process

yi = f(xi) + ε, ε ∼ N (0, σ20).

213

GPR allows to predict Gaussian estimates for any points x∗based on training examplesD := {(xi, yi)}Ni=1. We modelsimilarity in a local context of the state-action space bymeans of the Radial Basis kernel function. In regions thatare far away from training examples, large predicted vari-ance indicates high uncertainty in the estimate.

3.2 Expected DeviationIn our approach, we make extensive use of predicted un-certainty: From mean and variance for a state-action pairwe determine a measure of expected deviation from a givenvalue level. This deviation can be defined as either the ex-pected improvement or the expected degradation [10]. Weuse this measure to decide when an action is unsafe, or tofind promising actions during optimization.

4 Motion PrimitivesHumans intuitively recognize motions as looking human-like or being artificial. Yet it turns out to be surprisinglyhard to describe exactly which characteristics are respon-sible for this judgement and to apply these insights to robotmotions. Simple mathematical models usually are not suf-ficient and motions of contemporary robots often are de-scribed as being clumsy or jerky.On the other hand, it is generally accepted that human mo-tions are smooth. In particular, analyses of human reachingmovements show that these usually do not follow a straightline but rather proceed along a curve at varying speed.In order to evaluate our learning framework on humangrasping movements, we developed a controller that is

Figure 2: Generation of intermediate points along the de-sired trajectory. Dashed lines indicate the tangent lines atthe initial and final points PA and PE . The distances saand sE of the current location P towards PA and PE arescaled to obtain the auxiliary pointsHA, HE . The final ap-proach direction d is a linear combination of the directionsfrom P to the auxiliary points.

able to generate a wide range of anthropomorphic move-ments, which are determined by 31 parameters (see Ta-ble 1). This allows for a compact, low-dimensional repre-sentation of movements compared to the original trajecto-ries. The controller generates a trajectory which containsposes of the end-effector at regular intervals, as well as thedegree to which the hand is opened. Our controller is thusnot limited to a particular robot arm or design.

4.1 Trajectory GenerationThe first step in generating a movement is to determine avia point on the trajectory, in addition to the initial and fi-nal points. The trajectory is then split at this point to obtaintwo segments. For our purpose, the generation of graspingmovements, the highest point of the trajectory has provento be a suitable choice for the via point.The movement generation process consecutively processesthe trajectory’s segments, each in the same fashion. Everysegment is defined by its initial point PA and its final pointPE . Starting from the initial point, our controller gener-ates poses along the trajectory by successively moving to-wards a series of intermediate goal points. The generationof these intermediate points is illustrated in Figure 2.First, the direction towards the next intermediate point isdetermined by considering the tangent lines on the trajec-tory, passing through the segment’s initial and final points.These specify the desired direction of movement at thesepoints and are the same for all intermediate points of a seg-ment. To generate a smooth shift between these directions,an auxiliary point is chosen on each tangent line. Its posi-tion is determined by scaling the distance between the cur-rent position and each tangent’s boundary point accordingto eq. (1):

HA = PA + αA · ‖PA − P‖ · dAHE = PE − αE · ‖PE − P‖ · dE ,

(1)

where dA and dE represent the directions of the tangentlines and αA and αE are scaling factors. They are inde-pendent for every segment. The choice of the scaling fac-tors determines how much the generated trajectory is bent.That way, our controller allows for movements reachingfar back as well as for direct movements. The directiond towards the next intermediate point is formed by aver-aging the directions to the auxiliary points according to aweighting factor f :

d = f · HA − P‖HA − P‖

+ (1− f) · HE − P‖HE − P‖

. (2)

The value of f starts near 1 and gradually shifts towards0 as the movement generation progresses towards the finalpoint of the segment. At a certain distance, which can bedetermined by another input parameter θ, the influence ofthe initial direction vanishes completely and the directionto the next intermediate point is only determined by theauxiliary point on the final tangent. The weighting factor

214

f is determined by

f = 1− ‖PE − P‖θ · ‖PE − PA‖

; 0 ≤ f ≤ 1. (3)

The next pose P∗ is generated by moving along the deter-mined direction according to the current velocity v:

P∗ = P + v ·∆t · d. (4)

This value initially is zero and increases towards the viapoint where it reaches its maximum value vmax, whichalso is an input parameter to our controller. From the viapoint, the velocity decreases towards the final point. Thevelocity, however, is not a direct product of our controller.It only becomes apparent implicitly through the distance ofthe generated poses.To guarantee a smooth transition between the first and sec-ond segment of a movement, the final point and tangent ofthe first segment correspond to the first point and tangentof the second segment. By adding further via points andsegments, arbitrary trajectories can be generated.

Parameter Dimensionality

Initial point of the movement 3Via point 3Final point of the movement 3Direction at the initial point 3Direction at the via point 3Direction at the final point 3Opening degree of the hand 3Accelerations 2Limitation of the influence of the initial directions 2Scaling factors on tangents 4Offsets 2

Table 1: Parameters of our controller

4.2 Orientation and Opening of the HandJust like the position of the hand during a movement, itsorientation is a characteristic feature of anthropomorphicmovements. Our controller thus generates grasping trajec-tories that increasingly turn the hand towards the object tobe grasped as the hand approaches the object. In the courseof the first segment of a movement, the hand maintains aconvenient pose, independent of the object to be grasped.Once the via point has been crossed, the hand is graduallyturned towards the object to be grasped, by linearly inter-polating its orientation.Our controller also controls the degree to which the hand isopened according to three dedicated input parameters. Atthe beginning of the movement, the hand is closed. Whileit is moved to the via point, the hand is opened up to a de-gree that is determined by one of the three parameters. Thetwo remaining parameters determine the maximum degreeto which the hand will be opened and the distance to thetarget position, at which it should be attained. From thispoint on, the hand is closed until it grasps the object at thetarget position. This behaviour reflects the behaviour of

humans when grasping objects. Humans start to close thehand before actually reaching the desired object.

5 Interactive LearningIn our learning framework, we implement imitation andreinforcement learning as two alternative control flows.When the robot encounters a new situation, GPR providesGaussian estimates on the value of actions. Based onthis knowledge, our algorithm selects the adequate formof learning: In the case of insufficient knowledge aboutpromising actions, the robot acquires additional knowledgefrom human demonstration. Otherwise, it uses reinforce-ment learning to improve its actions.

5.1 Decision for Imitation or ReinforcementLearning

To make the decision for imitation or reinforcement learn-ing, we have to determine the best known action in thecurrent state scurr. In a first step, we search for a trainingexample x = (s, a) that has high value and that is close tothe current situation scurr by minimizing

x = argmin(s,a)∈X

(1− µ(s, a)) + ‖scurr − s‖2 . (5)

Next, we use the action a from this training example x toinitialize the search for the best known action abest in thecurrent state scurr. For this purpose, we maximize

abest = argmaxa

µ(scurr, a)− σ(scurr, a) (6)

through Rprop [11] gradient descent, which is faster thanstandard gradient descent and is less susceptible to end inlocal maxima. This optimization finds an action with largeexpected value but small uncertainty.Finally, we decide for a learning method according tothe expected degraded value [12] at the solution xbest :=(scurr, abest) towards Qbest := µ(xbest). If this indicator isbelow some threshold δ, there is no training example thatis sufficiently similar to the current situation and our al-gorithm asks for a human demonstration. Otherwise, thetraining examples contain enough information to predictthe outcome of actions and to safely use reinforcementlearning, without risking damage to the robot by execut-ing arbitrary movements.

5.2 Imitation LearningFor imitation learning, the user is asked to demonstrate themotion for the given situation. In doing so, his motionsare recorded using a motion capture rig and a data glove,yielding trajectory data for the human’s body parts. As ourrobot features a human-like joint configuration and upper-body proportions, the mapping from human to robot kine-matics is straightforward.

215

After segmentation, we extract the demonstrated actionfrom the recorded data in the form of a parameterized mo-tion primitive that is suitable as input to our controller. Forthis step, we apply two different techniques for the two dif-ferent categories of parameters of our controller.Parameters corresponding to geometric or kinematic fea-tures, such as the location of the first and last points of thetrajectory or the speed at a certain point, can be computeddirectly from the trajectory data using closed formulas thatonly depend on trajectory points. Out of the 31 parametersof our controller, the majority (23) belong to this group: theinitial, via, and final points, the corresponding directions,and the parameters that determine the degree to which thehand is opened. These parameters also allow for an intu-itive geometric interpretation.The remaining eight parameters cannot be computed inclosed form and thus need to be determined iteratively. Inthis work, we use the Downhill Simplex method by Nelderand Mead [13] to optimize a non-differentiable cost func-tion and to determine the optimal parameter values. Thecost function was designed to punish distance in space andtime between the human’s trajectory TD and the robot’strajectory TR(θ):

c(θ) = dist(TD, TR(θ)) + λ

(1− tD

tR(θ)

)2

. (7)

The variables tD and tR(θ) signify the duration of themovements and λ is a weighting factor. The distance mea-sure dist computes the mean squared distance between thetwo trajectories by averaging a point-to-line metric whichwas inspired by [14].Once all parameter values have been determined, the cor-responding movement is generated by the controller andevaluated by the robot. The resulting reward is stored,along with the computed parameter values, as a new train-ing example.

5.3 Reinforcement LearningIn reinforcement learning, we use Gaussian Process pre-diction to determine promising actions. To this end, wepropose an explorative optimization strategy that safelymaximizes value. We achieve this by trading off expectedimprovement and degradation. The chosen action alongwith the experienced reward eventually becomes a newtraining example that contributes to future predictions.Expected improvement optimization [10] selects pointsthat achieve highest expected improvement compared tothe current best value. The expected improvement consid-ers mean and variance of the GP posterior. Thus, the opti-mization strategy performs informed exploration which isbased on the knowledge gathered so far. We also incor-porate a lower bound on the expected degraded value intoour optimization criterion. By considering the expectedimprovement as well as the expected degradation, our ap-proach is able to produce an efficient search strategy that

also protects the robot from executing unpredictable ac-tions. Additionally, an adjustable lower bound on the ad-missible degraded value allows us to gradually focus thesearch on relevant parts of the search space with increas-ingly high performance.

6 Experiments

6.1 Experiment Setup

For our experiments, we used a robot with an anthropo-morphic upper body scheme [15]. In particular, its twoanthropomorphic arms have been designed to match theproportions and degrees of freedom (DoF) of their averagehuman counterparts. From trunk to gripper each arm con-sists of a 3 DoF shoulder, a 1 DoF elbow, and a 3 DoF wristjoint. This endows the robot with a workspace that is sim-ilar to that of humans, which greatly facilitates imitationlearning. Attached to the arm is a two-part gripper.In our setup, the robot is situated at a fixed position in frontof the table. The human teacher demonstrates graspingmotions on a second table.

6.2 Task Description

We define the task of the robot as grasping an object asprecisely as possible from arbitrary positions on the ta-ble. Thus, the state for our learning problem is the two-dimensional location of the object. The robot can measurethe location with its laser range finder. After the robot per-formed the grasping movement, we score its performanceby the displacement ∆ (in meters) of the object to its ini-tial location. Additionally, we penalize collisions, motionsoutside the workspace, and the missing of the object by thereward function

r(s, a) =

−1 collision or workspace failure,0 object missed,1−∆ otherwise.

(8)Obviously, this reward function does not meet the basic un-derlying assumption for GPs that any set of function valuesis jointly normal distributed. To still be able to apply ourapproach for this function, we apply GPR for each com-ponent and combine the predictions in a Gaussian mixturemodel.Throughout our experiments we set the kernel widths forthe object location to 0.2 for the successful and 0.02 forthe unsuccessful cases. For the motion parameters we setthe kernel widths to 0.02 and 0.002, respectively. We setthe decision threshold on the expected degraded value ofthe known best action to δ = 0.8.

216

1e-05

0.0001

0.001

0.01

0.1

1

10

0 50 100 150 200 250 300 350 400

combined

costs

number of iterations

Figure 3: Evolution of the combined costs (7) during theoptimization process, averaged over 30 example move-ments at different locations on the table.

-0.100.10.20.30.40.5

-0.3-0.2

-0.10

-0.8

-0.6

-0.4

-0.2

z

generated trajectorydemonstrated trajectory

xy

z

-0.200.20.40.6

-0.3-0.2

-0.10

-0.8

-0.6

-0.4

-0.2

z

demonstrated trajectorygenerated trajectory

xy

z

Figure 4: Left: Evaluated trajectories during optimizationfor a single movement. Right: Resulting trajectory.

6.3 Imitation Learning Experiments

In our approach, imitation learning is used to provide ini-tial actions when not enough information on executable ac-tions for the current location of the object are available. Insuch a situation, the robot asks for a demonstration. Thehuman teacher places the object at a similar location in hisworkspace and demonstrates the grasping movement (cf.Figure 1). Due to the generalization properties of our ap-proach, the objects do not have to be placed at exactly thesame position. From the recorded trajectory, we extract theparameters of a motion primitive as action. As a measureof imitation quality, we used the similarity in space andtime between the human demonstrations and the trajecto-ries generated by our algorithm. Out of the 31 parametersthat are required by our controller, 23 were determined di-rectly from the demonstrated trajectory. The remaining 8parameters were optimized iteratively.Figure 3 shows how the combined costs for spatial andtemporal dissimilarity decrease and finally converge to10−4 after 150 iterations. These results were obtainedby averaging the costs during the learning process for 30demonstrated grasps at different locations on the table.Figure 4 depicts how the reproduced trajectories approachthe demonstration during optimization. The locations ofthe first, the last and the highest point of the trajectoryare amongst the parameters that can be determined directlyand are thus fixed during optimization. Our method is alsorobust against errors in the motion capture data and fillsgaps smoothly in combination with our controller. In theend of the optimization process, the optimized trajectory is

visually indistinguishable from the demonstration.

6.4 Reinforcement Learning Experiments

For the task at hand, our learning approach has to compen-sate for kinematic differences between the human teacherand the robot. In our experiments, we thus choose four pa-rameters of the motion controller for optimization. Theseparameters determine offsets to the position of the grasp onthe table and the closure of the hand along the trajectory.The other parameters of our controller describe the anthro-pomorphic characteristics of the movement and hence donot affect reward. They are solely determined by humandemonstration. In simulation, the mapping between hu-man and robot motion is close to perfect. We thus add anartificial x-offset of 15cm to the grasping position parame-ters.

6.4.1 Evaluation of the Optimization Strategy

In a first simulation experiment, we limit the optimizationto the two offset parameters to visualize the learning strat-egy of our algorithm. We also keep the position of theobject fixed during the experiment. First, we demonstratea motion for the object location as a starting point for op-timization. Then, we apply our reinforcement learning ap-proach to reduce the error induced by the artificial offset.

6.4.2 Evaluation of the Optimization Strategy

Figure 5 (top) shows the evolution of reward in this exper-iment. Imitation learning already yields a grasping move-ment with a reward of 0.92. After about 30 reinforcementsteps, the optimization converges to a mean reward of ap-prox. 0.998 with standard deviation 0.0015, close to theoptimal reward.In Figure 5 (bottom), we visualize the adjustments madeto the offset parameters by our optimization strategy. Eachcircle represents an action chosen by our learning ap-proach.Filled blue circles indicate selected actions without ran-dom perturbation. Actions at unfilled red circles have beenfound after random reinitialization to escape potential lo-cal maxima. The optimization stategy proceeds in a goal-directed way towards the optimal parameters.Note that after larger steps, the strategy often takes smallersteps into the direction of known points. This is due to thefact, that the expected improvement is still large in regionsof the state space where GPR predicts high mean value andmedium variance. As soon as the uncertainty in these re-gions shrinks, our strategy proceeds towards new promis-ing regions. It is also interesting to note that close to theoptimal parameters, the optimization exhibits a star-shapedexploration strategy.

217

0.910.920.930.940.950.960.970.980.99

1

0 20 40 60 80 100 120 140 160 180 200

reward

number of iterations

0.060.08

0.10.12

0.140.16 -0.04

-0.03-0.02

-0.010

0.01

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1

rew

ard

with noisewithout noise

offset x

offset y

rew

ard

Figure 5: Evaluation of the optimization strategy for afixed object position in simulation. Top: Reward of theexecuted motion during optimization. Bottom: Proceedingof the optimization strategy. Filled blue circles indicateselected actions without random perturbation. Actions atunfilled red circles have been found after random reinitial-ization.

6.4.3 Reinforcement Learning with 4 Parameters onthe Real Robot

We repeated the experiment of the preceding section withthe real robot, this time learning all 4 parameters. The po-sition of the cup remained fixed. Figure 6 depicts the evo-lution of reward over time. It shows that the initialisationthrough an imitation already achieved a reward of 0.96 inthis experiment which was improved to 0.995 after 55 it-erations. This corresponds to a displacement of 5 mm. Incontrast to the experiments in the simulated enviroment,the reward does not increase further because of uncertain-ties in the robot kinematics which were not present in sim-ulation.

0.94

0.95

0.96

0.97

0.98

0.99

1

0 10 20 30 40 50 60 70 80 90 100

reward

number of iterations

Figure 6: Learning curve of the experiment using the realrobot. The object was placed at a fixed position 40cm infront of the robot and 20cm to its right.

0.35

0.4

0.45

0.5

0.1 0 -0.1 -0.2 -0.3 -0.4 -0.5

x

y

Figure 7: Top-down view of the robot’s entire workspace.The blue dots represent positions of the object during oursimulated experiment, where the robot asked for a demon-stration.

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1

0 200 400 600 800 1000

reward

number of iterations

0.360.390.420.450.48

-0.6-0.4

-0.200.2

0.920.930.940.950.960.970.980.99

1

reward

position x

position y

reward

0100200300400500

Figure 8: Top: Performance when learning graspingmovements within a 600cm2 area in simulation. Obtainedreward is shown in dark blue/gray. For values below theplotted range the robot failed to grasp the object and re-ceived zero reward. The light blue/gray line shows the me-dian reward within a window of 100 iterations. Bottom:Achieved reward for different object locations. The itera-tion count is encoded by the dots’ color. For clarity, onlythe first 500 iterations are plotted and failures to grasp theobject were omitted.

6.5 Interactive Learning ExperimentsFinally, we evaluate our combined approach to rein-forcement and imitation learning within the completeworkspace of the robot in simulation. The area has a sizeof 10x60cm. The robot was asked to grasp a object in ran-dom positions. It then had to decide whether to try to grasp

218

the object based on its experience or to demand a demon-stration. In the latter case, a human trainer put a cup at asimilar place in his/her one workspace and demonstratedthe desired movement. Figure 7 shows a top down viewof the robots workspace and the blue points indicate posi-tions at which the robot asked for demonstrations. All ofthese 15 demonstrations were required during the first 42iterations, 13 of them even within the first 24 iterations.Figure 8 shows the corresponding evolution of reward.Within 600 trials it achieves high performance for randomobject locations. Considering the large workspace and thenumber of different grasping movements this is a persua-sive result. The layered structure of the points is an in-dication of generalisation ability, as the reward increasessimultaneously all over the workspace.

7 Conclusion

In this work, we presented a new approach to intuitivelyand efficiently teach robots anthropomorphic movementprimitives. For this, we seamlessly integrate the wellknown paradigms of learning from demonstration and re-inforcement in a single coherent framework. In contrast toprevious approaches, that chained an imitation and a rein-forcement learning phase, our method implements them astwo alternative control paths. In every situation, the sys-tem decides which one to use to exploit each method’sstrengths while mitigating their shortcomings.To facilitate this, our approach relies on Gaussian Pro-cess Regression (GPR) to generalize a measure of rewardacross the combined state-action-space. From this, wecompute the expected deviation which is the basis uponwhich our system decides for imitation learning or rein-forcement learning. Furthermore, we use it to select ac-tions during reinforcement learning that trade off high re-ward versus predictable actions.To evaluate our approach, we considered the task of grasp-ing an object at an arbitrary position on a table. Ourimitation learning algorithm was able to produce move-ments that are visually indistinguishable from the demon-strations. The reinforcement strategy produced movementswith near-optimal reward in a goal-directed fashion, avoid-ing outlier movements that pose a threat to the robot. Ourcombined approach was able to teach the grasping task forarbitrary positions in a 600cm2 workspace to our robotfrom only 15 demonstrations.The advantages of our approach are manifold. First, theuse of learning methods that are known from human teach-ing psychology renders our method intuitively operableand helps in making robot programming accessible to aver-age users. Second, our unique combination of both meth-ods allows for flexible learning that uses the most appropri-ate method in every situation. This way we achieve data-efficient learning and circumvent shortcomings of the indi-vidual methods.

References[1] A. Billard, S. Calinon, R. Dillmann, and S. Schaal.

Robot Programming by Demonstration. In Handbookof Robotics. Springer, 2008.

[2] G. Biggs and B. Macdonald. A survey of robot pro-gramming systems. In Proc. of the Australasian Conf.on Robotics and Automation, CSIRO, 2003.

[3] S. Calinon, F. Guenter, and A. Billard. On Learning,Representing and Generalizing a Task in a HumanoidRobot. IEEE Trans. on systems, man and cybernetics,2007.

[4] L. Pack Kaelbling, M. L. Littman, and A. W. Moore.Reinforcement learning: A survey. J. of artificial in-telligence research, 1996.

[5] N. Kohl and P. Stone. Policy Gradient ReinforcementLearning for Fast Quadrupedal Locomotion. In Proc.of the IEEE Int. Conf. on Robotics and Automation(ICRA), 2004.

[6] D. J. Lizotte, T. Wang, M. H. Bowling, and D. Schu-urmans. Automatic Gait Optimization with GaussianProcess Regression. In Proc. of the Int. J. Conf. onArtificial Intelligence, 2007.

[7] S. Schaal. Learning From Demonstration. Advancesin neural information processing systems, 1997.

[8] F. Guenter and A. Billard. Using ReinforcementLearning to Adapt an Imitation Task. In Proc. of theIEEE/RSJ Int. Conf. on Intelligent Robots and Sys-tems, 2007.

[9] C. E. Rasmussen and C. K. I. Williams. GaussianProcesses for Machine Learning. MIT Press, 2006.

[10] D. R. Jones. A Taxonomy of Global OptimizationMethods Based on Response Surfaces. J. of GlobalOptimization, 2001.

[11] M. Riedmiller. Rprop-description and implementa-tion details. Technical report, University of Karls-ruhe, 1994.

[12] K. Gräve. Lernen von Greifbewegungen aus Imita-tion und eigener Erfahrung. Master’s thesis, Univer-sität Bonn, November 2009.

[13] J. A. Nelder and R. Mead. A Simplex Methodfor Function Minimization. The Computer Journal,7(4):308–313, January 1965.

[14] A. Censi. An ICP variant using a point-to-line met-ric. In Proc. of the IEEE Int. Conf. on Robotics andAutomation (ICRA), 2008.

[15] J. Stückler and S. Behnke. Integrating Indoor Mobil-ity, Object Manipulation and Intuitive Interaction forDomestic Service Tasks. In Proc. of the IEEE-RASInt. Conf. on Humanoid Robots, 2009.

219