Hierarchical Learning of Motor Skills with Information-Theoretic Policy Search

Gerhard Neumann1 and Christian Daniel1 and Andras Kupcsik2 andMarc Deisenroth1 and Jan Peters1, 3

1 Intelligent Autonomous Systems, TU Darmstadt2 Department of Electrical and Computer Engineering, National University of Singapore

3 Max Planck Institute for Intelligent Systems

Abstract

The key idea behind information-theoretic policy search isto bound the ‘distance’ between the new and old trajectorydistribution, where the relative entropy is used as ‘distancemeasure’. The relative entropy bound exhibits many bene-ficial properties, such as a smooth and fast learning processand a closed-form solution for the resulting policy. In thispaper we will summarize our work on information theoreticpolicy search for motor skill learning where we put particu-lar focus on extending the original algorithm to learn severaloptions for a motor task, select an option for the current situa-tion, adapt the option to the situation and sequence options tosolve an overall task. Finally, we illustrate the performanceof our algorithm with experiments on real robots.

IntroductionPolicy search seeks a parameter vector ω of a parametrizedpolicy π that yields high expected long-term rewardRω . Despite of recent successful applications of policysearch [19, 22, 13, 14] on real robots, most of these appli-cations have been limited to learning a single movement inisolation.

In order to overcome this limitation, a more powerfulframework is needed which can learn several options forsolving a task, select between these options, adapt the in-dividual options based on the current situation, and, finally,sequence options in order to solve an overall, more complextask. In our recent work on information-theoretic motor skilllearning we developed an integrated framework which cancope with all these requirements in a unified way.

Our algorithm is based on the Relative Entropy PolicySearch Algorithm (REPS) proposed by Peters et. al. [18].The main insight used by REPS is that the relative entropybetween the trajectory distributions of two subsequent poli-cies during policy search should be bounded. This boundensures a smooth and fast learning progress in policy search[19, 4]. While such insight has already been used by the nat-ural policy gradient (NPG) algorithms [19], NPG algorithmsonly approximate this bound and require a user-specifiedlearning rate. In contrast, REPS does not require such anapproximation of the relative entropy nor a user specifiedlearning rate.

We employ an hierarchical approach where we learn anupper level policy π(ω) which represents a search distribu-

tion over the parameters ω of the policy π(u|x,ω). Af-ter choosing the parameter vector ω at the beginning of anepisode with the upper-level policy, the lower-level policyπ(u|x;ω) determines the controls u during the episode,where x denotes the state of the robot. We first considerthe contextual policy search scenario where policies are gen-eralized to multiple contexts. The context s contains taskvariables that are specified before task execution. For ex-ample, the context might contain the objectives of the robot,such as a target location of the end-effector, or properties ofthe environment such as a mass to lift. To generalize thelower-level policy to multiple contexts si, we will use a con-ditional distribution π(ω|s) as upper level policy, resultingin the contextual REPS algorithm. We further improved thelearning speed of the contextual REPS algorithm by endow-ing our algorithm with learned Gaussian Process [20, 8] for-ward models [15].

As the next step, we extend the REPS framework to learn-ing several options for a motor task [4, 5]. Learning severaloptions is useful as for a different context, a different type ofmovement need to be selected, e.g., choosing between fore-hand and backhand in a tennis play and also increases therobustness of the learned policy. To achieve this goal, weadd an additional layer of hierarchy to the upper level pol-icy, which is now composed of a gating policy π(o|s), whichselects the option based on the current context and the indi-vidual option policies π(ω|s, o) which select the parametervector to execute.

Many motor tasks, such as a tennis play or locomotion re-quire a sequence of options to be executed in order to fulfillthe overall task. For example, in a tennis play an option rep-resents a forehand or backhand swing and typically manyswings are needed to win a tennis game. Hence, we needa framework which can learn to sequence options such thatthe options maximize the long-term reward of the agent incombination. We extend our framework of learning an hier-archical policy to the multi-step case, where the upper levelpolicy has to choose new policy parametersω every time thelower level policy is terminated.

We illustrate the properties of our learning framework onthree different tasks. In the robot-tetherball task the robothas to learn two different solutions how to hit a ball around apole. In the robot-hockey target hitting task we want to shoota puck as accurately as possible such that we hit another

puck, the target puck. The target puck has to move for acertain distance. Finally, we extend the robot hockey task toseveral target zone which require a sequence of shoots.

Relative Entropy Policy Search for LearningUpper-Level Policies

In this section we will first state the problem of learning anupper-level policy for contextual policy search, which re-sults in the contextual REPS algorithm. We will furtherextend this algorithm with learned forward models to im-prove learning speed, add an additional layer of hierarchy tochoose between several options and finally extend the REPSframework to learning to select multiple options sequentiallyin an finite horizon formulation.

Contextual Policy SearchContextual policy search [10, 17] aims to adapt the parame-ters ω of a lower-level policy to the current context s. Thecontext contains task variables that are specified before taskexecution but are relevant for determining the lower-levelpolicy for the task. The aim of contextual policy learning isto learn an upper-level policy π(ω|s) that chooses parame-ters ω according to the current context s that maximize theexpected reward

J = Es,ω[Rsω] =∫s,ω

µ(s)π(ω|s)Rsωdsdω.

The distribution µ(s) specifies the distribution over the con-texts and

Rsω = Eτ [r(τ , s)|s,ω] =∫τ

p(τ |s,ω)r(τ , s), (1)

is the expected reward of the whole episode when using pa-rameters ω for the lower-level policy π(u|x,ω). in contexts. In (1), τ denotes a trajectory, p(τ |s,ω) a trajectory dis-tribution, and r(τ , s) a user-specified reward function.

Contextual REPS. The basic idea of Relative EntropyPolicy Search [18] is to bound the relative entropy betweenthe old and new trajectory distribution. In the case of contex-tual policy search, we will abstract the trajectory distributionby a distribution p(s,ω) = π(ω|s)p(s) over the context sand parameter ω.

Hence, contextual REPS [15] is defined as optimizationproblem in the space of the distribution p(s,ω). Here, wewant to maximize the expected reward

J =∑

s,ωp(s,ω)Rsω, (2)

with respect to p while bounding the relative entropy to thepreviously observed distribution q(s,ω), i.e.,

ε ≥∑

s,ωp(s,ω) log

p(s,ω)

q(s,ω). (3)

As the context distribution p(s) =∑ω p(s,ω) is specified

by the learning problem and given by µ(s), we also need toadd the constraints

∀s : p(s) = µ(s)

to the REPS optimization problem. However, to fulfill theseconstraints also for continuous context vectors s, we matchfeature averages instead of single probability values, i.e.,∑

sp(s)φ(s) = φ, (4)

where φ(s) is a feature vector describing the context andφ is the mean observed feature vector. Furthermore, we re-quire that the estimated p(s,ω) defines a distribution, i.e.,∑s,ω p(s,ω) = 1.The resulting optimization problem yields the closed-

form solution

p(s,ω) ∝ q(s,ω) exp(Rsω−V (s)

η

)(5)

where V (s) = θTφ(s) is a value function [18]. The param-eters η and θ are Lagrangian multipliers used for the con-straints (3) and (4), respectively. These parameters are foundby optimizing the dual function g(η,θ) [4]. The functionV (s) emerges from the feature constraint in (4) and servesas context dependent baseline for the reward signal.

Estimating the Upper-Level Policy. The optimization de-fined by the REPS algorithm is only performed on a discreteset of samples D = {s[i],ω[i],R[i]

sω}, i = 1, . . . , N . The re-sulting probabilities p(s[i],ω[i]) of these samples are usedto weight the samples. The upper-level policy π(ω|s) issubsequently obtained by performing a weighted maximum-likelihood estimate. We use a linear-Gaussian model to rep-resent the upper-level policy π(ω|s), i.e.,

π(ω|s) = N (ω|a+As,Σ). (6)

In this example, the parameters θ of the upper-level policyare given as {a,A,Σ}.

Improving Data-Efficiency with Learned ForwardModelsModel-free policy search algorithms may often require anexcessive amount of roll-outs even for simple tasks. Model-based approaches learn a forward model of the robot and itsenvironment [7, 2, 21, 1] and use the model to predict thelong term reward and typically outperform model-free pol-icy search algorithms considerably if a good model can belearned. We incorporated the idea of using learned mod-els into the contextual REPS algorithm [15] by using thelearned models as simulator to create artificial data points.As model-based approaches are sensitive to model errors[1], we use Gaussian Process models to represent the un-certainty of the learned model itself, and, hence, reduce theeffect of model errors [7].

With data from the real robot, we learn the forward mod-els of the robot and its environment. Moreover, we estimatea distribution µ(s) over the contexts, such that we can eas-ily generate new context samples s[j]. Subsequently, wesample a parameter vector ω[j] and use the learned prob-abilistic forward models to sample L trajectories τ [l]

j forthe given context-parameter pair. Finally, we obtain ad-ditional artificial samples (s[j],ω[j],R[j]

sω), where R[j]sω =∑

l r(τ[l]j , s

[j])/L, that are used for updating the policy.

Learning Versatile Motor Skills with HierarchicalREPSIn order to select between several options o, we extend theupper-level policy with an additional layer of hierarchy. Thehierarchical upper-level policy is now given by

π(ω|s) =∑o

π(o|s)π(ω|s, o), (7)

where we will denote π(o|s) as gating and π(ω|s, o) as op-tion policy. Such hierarchy has two main advantages. First,the gating policy allows for choosing different options in dif-ferent contexts which again enables us to use rather simpleoption policies, such as a linear Gaussian models. Second,the gating policy can be used to learn multiple solutions forthe same context, increasing the versatility of the learnedmotor skill.

We reformulate the problem of estimating the hier-archical policy given in (7) as latent variable estima-tion problem, i.e., we treat the options o as unobservedvariables. As in the standard REPS formulation, webound the Kullback-Leibler divergence between q(s,ω) andp(s,ω) =

∑o π(ω|s, o)π(o|s)µπ(s). However, using the

marginal p(s,ω) in the bound would no longer allow for aclosed form solution of p(s,ω, o). Thus, we need to usean iterative update rule, which is strongly inspired by theExpectation-Maximization algorithm [16]. This update re-sults from the bound∑

s,ω

∑o

p(s,ω, o) logp(s,ω, o)

q(s,ω)p(o|s,ω)≤ ε, (8)

where p is a proposal distribution, often called responsibilityin EM-based algorithms. In the E-step of our algorithm, theresponsibilities are determined by

p(o|s,ω) = p(s,ω, o)∑o p(s,ω, o)

. (9)

In the M-step, the responsibilities p(o|s,ω) are kept fixedand we directly optimize for p(s,ω, o). Both iterations, theE- and the M-step, can be proven to increase a lower boundof the original optimization problem [4]. Thus, REPS withlatent variables converges to a local optimum of the originalcontextual REPS optimization problem.

As we are interested in versatile solutions, we also want toavoid that several options concentrate on the same solution.Hence, we want to limit the ‘overlap’ between options in thestate-action space. In order to do so, we additionally boundthe expected entropy of the responsibilities π(o|s,ω), i.e.,

−∑s,ω

p(s,ω)∑o

π(o|s,ω) log π(o|s,ω) ≤ κ. (10)

Low entropies of π(o|s,ω) ensures that our options do notoverlap and, thus, represent individual and clearly separatedsolutions. We can again replace π(o|s,ω) with the respon-sibilities p(o|s,ω) in the term log π(o|s,ω).

By combining the constraints given in Equations (8) and(10) with the original optimization problem of contextualREPS, we result in a hierarchical version of REPS, denoted

as HiREPS [4]. We can again determine a closed form solu-tion for p(s,ω, o) which is given by

p(s,ω, o) ∝ q(s,ω)p(o|s,ω)1+ξ/η exp(Rsω − V (s)

η

).

(11)The parameter ξ is the Lagrange multiplier for the entropybound of the responsibilities given in Equation (10). Notethat the hierarchical approach reduces to contextual REPS ifwe only use one option.

Policy Learning for Sequential Motor TasksIn many learning tasks we need to execute multiple op-tions in sequence. To do so, we reformulated the HiREPSframework to be applicable to learning a sequence of op-tions [6]. We formulate the problem of sequencing optionsas a Markov Decision Process (MDP). Each option oi is exe-cuted for di seconds. This duration can also be parametrizedby ω and, hence, can be adapted by the agent, however, sofar, we always use a pre-specified duration of the options.

The execution of an option in context s results in a transi-tion to context s′ with probability Pωss′ = p(s′|s,ω) and anexpected reward of Rsω = r(s,ω) of the option. We con-centrate on the finite-horizon case, i.e., each episode con-sists of K steps where each step is defined as the execu-tion of an individual option. In this case, the optimal policyπk(ω|sk) =

∑o πk(s|o)πk(ω|o, s) and its state distribu-

tion µπk (s) depends on the decision time step k. Our task isto learn a policy π(ω|s) which maximizes the expected totalreward

J = E[rK(sK+1) +

∑K

k=1rk(sk,ωk)

], (12)

where the expectation is performed over the states s1:K+1

and the selected parameters ωk ∼ πk(ω|sk). The termrK(sK+1) denotes the final reward for ending up in the statesK+1 after executing the option.

Time-Indexed HiREPS. In order to optimize a sequenceof options instead of the execution of a single option, wehave to find a policy which maximizes the average reward

J =∑s

µπK+1(s)rK(s) +

K∑k=1

∑s,ω

µπk (s)πk(ω|s)Rksω,

(13)where Rksω = rk(s,ω).

We also bound the distance between pk(s,ω) and the ob-served distribution of the old policy qk(s,ω), as well as theentropy of the responsibilities pk(o|s,ω). Thus, the mainprinciple remains unchanged from the episodic case. How-ever, these bounds are now applied to each decision step k.

The most important difference to the episodic case consid-ered previously lies in the application of the state transitionconstraints for the state distribution µπk (s). In our finite hori-zon MDP, the state-distributions µπk (s) have to be consistentwith the policy πk(ω|s) and the transition model Pωss′ , i.e.,

∀s′, k : µπk (s′) =

∑s,ω

µπk−1(s)πk−1(ω|s)Pωss′ , (14)

for each step of the episode. These constraints connect thepolicies for the individual time-steps and result in a policyπk(ω|s) that optimizes the long-term reward instead of theimmediate ones. As in the episodic case, these constraintsare again implemented by matching feature averages. Thisformulation requires the use of a model Pωss′ which needsto be learned from data. However, for simplicity, we willapproximate this model by the single-sample outcome s′ ofexecuting the parameters ω in state s. This approximation isvalid as long as the stochasticity in our system is low. Learn-ing the model Pωss′ is part of future work.

The closed form solution of the joint distributionpk(s,ω, o) yields

pk(s,ω, o) ∝ qt(s,ω)pk(o|s,ω)1+ξk/ηk exp(Aksωηk

),

(15)where Aksω is given by the term

Aksω = Rksω + Ep(s′|s,ω)[Vk+1(s′)]− Vk(s).

Due to our time-indexed formulation, we now obtain onevalue function Vk(s) = φ(s)Tθk per time step. In addition,the scaling factor ηk and the ξk factor are time dependent.

We can see that the reward Rksω is transformed into anadvantage function Aks,ω where the advantage function nowalso depends on the expected value of the next state. Theadvantage function emerged naturally from the formulationof the constraint in Eq. (14).

For the state features, we will use radial basis function(RBF) features located at the sample points for all our ex-periments. We use a Gaussian gating for the gating policyπk(o|s) and a Gaussian linear model for the parameter se-lection policy πk(ω|s, o).

ResultsAs the lower-level policy needs to scale to anthropomorphicrobotics, we implement them using the Dynamic MovementPrimitive [9] approach, which we will now briefly review.

Dynamic Movement PrimitivesTo parametrize the lower-level policy we use an exten-sion [11] to Dynamic Movement Primitives (DMPs). ADMP is a spring-damper system that is modulated by a forc-ing function f(z;v) = Φ(z)Tv. The variable z is the phaseof the movement. The parameters v specify the shape of themovement and can be learned by imitation. The final posi-tion and velocity of the DMP can be adapted by changingadditional hyper-parameters yf and yf of the DMP. The ex-ecution speed of the movement is determined by the timeconstant τ . For a more detailed description of the DMPframework we refer to [11].

Learning Versatile Solutions for Robot TetherballIn robot tetherball, the robot has to shoot a ball that is fixedwith a string on the ceiling such that it winds around a pole.We mounted a table-tennis paddle to the end-effector of therobot arm. The robot obtains a reward proportional to thespeed of the ball winding around the pole. There are two

different solutions, to wind the ball around the left or to theright side of the pole. Two successful hitting movements ofthe real robot are shown in Figure 1.

We decompose our movement into a swing-in motion anda hitting motion. For both motions we extract the shape pa-rameters v of the DMPs by kinesthetic teach-in [3]. As weused the non-sequential algorithm for this experiment, werepresented the two motions by a single set of parametersand jointly learn the parameters for the two DMPs. For bothmovements, we learn the final positions yf and velocitiesyf of all seven joints. Additionally, we learn the waitingtime between both movements. This task setup results ina 29-dimensional parameter space for our robot Tetherballtask.

The reward is determined by the speed of the ball whenthe ball winds around the pole. We define winding aroundthe pole as the ball passing the pole on the opposite sidefrom the initial position.

We initialize our algorithm with 15 options and sample15 trajectories per iteration. The learning curve is shown inFigure 3(a). The noisy reward signal is mostly due to thevision system (Microsoft Kinect) and partly also due to realworld effects such as friction. Two resulting movements ofthe robot are shown in Figure 1.

Robot Hockey Target ShootingIn this task we used GPREPS to learn how to shoot hockeypucks. The objective was to make a target puck move aspecified distance. Since the target puck was out of reach,it could only be moved indirectly by shooting a secondhockey puck at the target puck. The initial location [bx, by]

T

of the target puck was varied in both dimensions as wellas the distance d∗ the target puck had to be shoot, i.e.,s = [bx, by, d

∗]T . The simulated robot task is depicted inFig. 2.

The agent had to adapt the final positions yf , velocitiesyf and the time scaling parameter τ of the DMPs, resultingin a 15-dimensional lower-level policy parameter ω. The re-ward function was given as the negative minimum squareddistance of the robot’s puck to the target puck during a play.In addition, we penalized the squared difference between thedesired travel distance and the distance the target puck actu-ally moved.

GPREPS first learned a forward model to predict the ini-tial position and velocity of the first puck after contact withthe racket and a travel distance of 20 cm. Subsequently,GPREPS learned the free dynamics model of both pucks andthe contact model of the pucks. For the contact model, weassumed a known radius of the pucks to detect contact ofthe pucks. Based on the position and velocities of the pucksbefore contact, the velocities of both pucks were predictedafter contact with the learned model.

We compared GPREPS to directly predicting the rewardRsω , model-free REPS and CrKR [12], a state-of-the-artmodel-free contextual policy search method. The resultinglearning curves are shown in Fig. 3(b). GPREPS learnedthe task already after 120 interactions with the environ-ment while the model-free version of REPS needed approx-imately 10000 interactions. Moreover, the policies learned

Figure 1: Time series of a successful swing of the robot. The robot first has to swing the ball to the pole and can, subsequently,when the ball has swung backwards, arc the ball around the pole. The movement is shown for a shoot to the left and to the rightof the pole.

by model-free REPS were of lower quality. Directly predict-ing the rewards from parameters ω using a single GP modelresulted in faster convergence but the resulting policies stillshowed a poor performance (GP direct). The results showthat CrKR could not compete with model-free REPS.

The learned movement is shown in Fig. 2 for a specificcontext. After 100 evaluations, GPREPS placed the targetpuck accurately at the desired distance with a displacementerror ≤ 5 cm.

Finally, we evaluated the performance of GPREPS on thehockey task using a real KUKA lightweight arm, which isshown in Figure 3(c). A Kinect sensor was used to trackthe position of the two pucks at a frame rate of 30Hz. Thedesired distance d∗ to move the target puck ranged between0 m and 0.6 m.

Sequential Robot HockeyWe used the sequential robot hockey task to evaluate oursequential motor skill learning framework. In the sequentialrobot hockey task, the robot has to move the target puck intoone of three target areas. The target areas are defined by a

specified distance to the robot. The first zone is defined asdistance from 1.4 to 1.8m, the second zone from 1.8 to 2.2mand the last zone from 2.2 to 2.6m. The robot gets rewardsof 1, 2, and 3 for reaching zone 1, 2 or 3, respectively, withthe target puck. The reward is only given after each episodewhich consists of three shots of the robot’s puck. After eachshot, the control puck is returned to the robot. The targetpuck, however, is only reset after each episode, i.e., everythird shot. The setup of the robot hockey task is shown inFigure 4(a).

The 2-dimensional position of the target puck defines thecontext of the task. After performing one shot, the agentobserves the new position of the target puck to plan the sub-sequent shot. In order to give the agent an incentive to shootat the target puck, we punished the agent with the negativeminimum distance of the control puck to the target puck af-ter each shot. While this reward was given after every step,the zone reward was only given at the end of the episode(every third step) as rK+1(sK+1).

We choose the same DMP representation as for the previ-ous experiment but excluded the time scaling factor τ from

[bx,by]

d*

Figure 2: Robot hockey target shooting task. The robot shot apuck at the target puck to make the target puck move for a specifieddistance. Both, the initial location of the target puck [bx, by]

T andthe desired distance d∗ to move the puck were varied. The contextwas s = [bx, by, d

∗].

Iteration

Rew

ard

Real Robot Learning Curve

0 5 10 15 20 25 30 35 40 45 50-1800-1600-1400-1200-1000

-800-600-400-200

0200

HiREPS

(a)

evaluations

rew

ard

101 102 103 104

-0.6

-0.4

-0.2

0

GPREPSREPSREPS (GP direct)CRKR

(b)

45 50 55 60 65 70 75 80−0.14

−0.12

−0.1

−0.08

−0.06

−0.04

evaluations

rew

ard

GPREPS

(c)

Figure 3: (a) Average rewards for learning tetherball on the real robot. Mean and standard deviation of three cross validations.In all of the three trials, after 50 iterations the robot has found solutions to wind the ball around the pole on either side. (b)Learning curves on the robot hockey task in simulation. GPREPS was able to learn the task within 120 interactions with theenvironment, while the model-free version of REPS was not able to find high-quality solutions. (c) GPREPS learning curve onthe real robot arm.

(a)# Episodes

Ave

rage

Rew

ard

0 500 1000 1500

0

0.5

1

1.5

2

2.5

3

EpisodicSequential

(b)Episodes

Rew

ard

0 50 100 150 200 250 300-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3 Real Robot Hockey

(c)

Figure 4: (a) The sequential robot hockey task. The robot has two pucks, the pink control puck and the yellow target puck. The task is toshoot the yellow target puck into one of the colored reward zones. Since the best reward zone is too far away from the robot to be reachedwith only one shot, each episode consists of three strikes. After each strike the control puck is returned to the robot, but the target puck is onlyreset after one episode is concluded. (b) Comparison of sequential motor primitive learning to the episodic learning setup on the simulatedrobot hockey task. The sequential motor primitive learning framework was able to find a good strategy to place the puck in the third rewardzone in most of the cases while the episodic learning scenario failed to learn such a strategy. (c) One trial of of the real robot hockey tasks.The robot starts with a negative initial reward and learns to achieve an average reward of 2.5 after 300 episodes.

the parameter vector of a single option. We also comparedto the episodic policy learning setup with REPS, where weused one option to decode all three shots, resulting in a42 dimensional parameter vector. In contrast, the time in-dexed HiREPS had to optimize three policies with one 14-dimensional parameter space each.

We compared our sequential motor primitive learningmethod with its episodic variant on a realistic simulation.The comparison of both methods can be seen in Figure 4(b).The episodic learning setup failed to learn a proper policywhile our sequential motor primitive learning frameworkcould steadily increase the average reward. Our methodreached an average reward of 2.3 after learning for 1500episodes. Note that an optimal strategy would have reacheda reward value of 3, however, this is still a clear improve-ment in comparison to the episodic setup, which reached afinal reward value of 1.4.

On the real robot, we could reproduce the simulation re-sults. The robot learned a strategy which could move the tar-get puck to the highest reward zone in most of the cases after300 episodes. The learning curve is shown in Figure 4(c).During learning the robot steadily adapted his strategy whenit mastered the necessary motor skills to achieve higher re-wards by placing the target puck in the highest reward zones.

ConclusionWe identified three main challenges for motor skill learning— learning to adapt options to the current context, learningto select an option based on the current context and learningto sequence options.

We presented a unified approach which tackles these chal-lenges in an integrated way. The information theoreticformulation of REPS not only ensures a smooth learningprogress, it also allows for mathematically sound extensionsof REPS which can be incorporated as constraints to the op-timization problem. Our future work will concentrate onlearning more complex hierarchies and incorporating newtypes of learned forward models in our algorithm.

References[1] C. G. Atkeson and J. C. Santamarıa. A Comparison of

Direct and Model-Based Reinforcement Learning. InInternational Conference on Robotics and Automation(ICRA), 1997.

[2] J. A. Bagnell and J. G. Schneider. Autonomous He-licopter Control using Reinforcement Learning Pol-icy Search Methods. In Proceedings of the In-ternational Conference for Robotics and Automation(ICRA), pages 1615–1620, 2001.

[3] H. Ben Amor, E. Berger, D. Vogt, and B. Jung. Kines-thetic bootstrapping: Teaching motor skills to hu-manoid robots through physical interaction. KI 2009:Advances in Artificial Intelligence, pages 492–499,2009.

[4] C. Daniel, G. Neumann, and J. Peters. HierarchicalRelative Entropy Policy Search. In International Con-ference on Artificial Intelligence and Statistics (AIS-TATS), 2012.

[5] C. Daniel, G. Neumann, and J. Peters. LearningConcurrent Motor Skills in Versatile Solution Spaces.In IEEE/RSJ International Conference on IntelligentRobots and Systems, 2012.

[6] C. Daniel, G. Neumann, and J. Peters. Learning se-quential motor tasks. submitted, 2012.

[7] M. P. Deisenroth, C. E. Rasmussen, and D. Fox. Learn-ing to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning. In Proceedings ofR:SS, 2011.

[8] Marc Peter Deisenroth and Carl Edward Rasmussen.PILCO: A Model-Based and Data-Efficient Approachto Policy Search. In 28th International Conference onMachine Learning (ICML), pages 465–472, 2011.

[9] Auke Jan Ijspeert and Stefan Schaal. Learning Attrac-tor Landscapes for Learning Motor Primitives. In Ad-vances in Neural Information Processing Systems 15,(NIPS). MIT Press, Cambridge, MA, 2003.

[10] J. Kober and J. Peters. Policy Search for Motor Prim-itives in Robotics. Machine Learning, pages 1–33,2010.

[11] Jens Kober, Katharina Mulling, Oliver Kroemer,Christoph H. Lampert, Bernhard Scholkopf, and JanPeters. Movement Templates for Learning of Hittingand Batting. In International Conference on Roboticsand Automation (ICRA), 2010.

[12] Jens Kober, Erhan Oztop, and Jan Peters. Reinforce-ment Learning to adjust Robot Movements to New Sit-uations. In Proceedings of the Robotics: Science andSystems Conference (RSS), 2010.

[13] N. Kohl and Peter Stone. Policy Gradient Reinforce-ment Learning for Fast Quadrupedal Locomotion. InInternational Conference for Robotics and Automation(ICRA), 2003.

[14] Petar Kormushev, Sylvain Calinon, and Darwin GCaldwell. IEEE/RSJ International Conference on In-telligent Robots and Systems. In Proceedings ofthe IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS), 2010.

[15] A. Kupsic, M. P. Deisenroth, J. Peters, and G. Neu-mann. Data-efficient contextual policy search for robotmovement skills. submitted, 2012.

[16] Radford Neal and Geoffrey E. Hinton. A View OfThe Em Algorithm That Justifies Incremental, Sparse,And Other Variants. In Learning in Graphical Models,pages 355–368. Kluwer Academic Publishers, 1998.

[17] G. Neumann. Variational Inference for Policy Searchin Changing Situations. In Proceedings of the 28th In-ternational Conference on Machine Learning, (ICML2011), pages 817–824, New York, NY, USA, June2011. ACM.

[18] J. Peters, K. Mulling, and Y. Altun. Relative EntropyPolicy Search. In Proceedings of the 24th NationalConference on Artificial Intelligence (AAAI). AAAIPress, 2010.

[19] Jan Peters and Stefan Schaal. Reinforcement Learningof Motor Skills with Policy Gradients. Neural Net-works, (4):682–97, 2008.

[20] C. E. Rasmussen and C. K. I. Williams. Gaussian Pro-cesses for Machine Learning. The MIT Press, 2006.

[21] J. G. Schneider. Exploiting Model Uncertainty Esti-mates for Safe Dynamic Control Learning. In NIPS,pages 1047–1053. 1997.

[22] E. Theodorou, J. Buchli, and S. Schaal. Reinforce-ment Learning of Motor Skills in High Dimensions:a Path Integral Approach. In Robotics and Automa-tion (ICRA), 2010 IEEE International Conference on,pages 2397–2403, 2010.