Online Feature Selection for Model-based Reinforcement Learning

Trung Thanh Nguyen nttrung@comp.nus.edu.sgZhuoru Li lizhuoru@comp.nus.edu.sgTomi Silander silander@comp.nus.edu.sgTze-Yun Leong leongty@comp.nus.edu.sg

School of Computing, National University of Singapore, Singapore, 117417

Abstract

We propose a new framework for learningthe world dynamics of feature-rich environ-ments in model-based reinforcement learn-ing. The main idea is formalized as a new,factored state-transition representation thatsupports efficient online-learning of the rel-evant features. We construct the transitionmodels through predicting how the actionschange the world. We introduce an onlinesparse coding learning technique for featureselection in high-dimensional spaces. We de-rive theoretical guarantees for our frameworkand empirically demonstrate its practicalityin both simulated and real robotics domains.

1. Introduction

In model-based reinforcement learning (RL), factoredstate representations, often in the form of dynamicBayesian networks (DBNs), are deployed to exploitstructures of the world dynamics. This allows theagent to plan and act in large state spaces withoutactually visiting every state. However, learning theworld dynamics of a complex environment is very dif-ficult and often computationally infeasible. Most re-cent work in this area is based on the RMAX frame-work (Brafman & Tennenholtz, 2003), and focuses onsample-efficient learning of the optimal policies. Thisapproach incurs heavy computational costs for maxi-mizing information gain from every interaction, evenin carefully designed, low-dimensional spaces.

We propose a variant formulation of the factoredMarkov decision process (MDP) that incorporates aprincipled way to compactly factorize the state space,

Proceedings of the 30 th International Conference on Ma-chine Learning, Atlanta, Georgia, USA, 2013. JMLR:W&CP volume 28. Copyright 2013 by the author(s).

while capturing comprehensive transition and rewarddynamics information. We also propose an onlinemultinomial logistic regression method with grouplasso to automatically learn the relevant structure ofthe world dynamics model. While the regression mod-els cannot capture the full conditional distributionslike DBNs, their simplicity allows fast, online learn-ing in very high dimensional spaces. Online featureselection is implemented with operating the regressionalgorithm in our variant MDP formulation.

In the rest of the paper, we will first introduce anMDP representation that captures the world dynamicsas action effects. We then present an online learningalgorithm for identifying relevant features via sparsecoding, and show that in theory our framework shouldlead to computationally efficient learning of a near op-timal policy. Due to the space limit, full proofs areplaced in the supplementary materials. To back upthe theoretical claims, we conduct experiments in bothsimulated and real robotics domains.

2. Method

In RL, a task is typically modelled as an MDP definedby a tuple (S,A, T,R, γ), where S is a set of states; Ais a set of actions; T : S×A×S → [0, 1] is a transitionfunction, such that T (s, a, s′) = P (s′|s, a) specifies theprobability of transiting to state s′ upon taking an ac-tion a at state s; R is a reward function indicatingimmediate expected reward after the state transitions

a−→ s′; and future reward that occurs t time stepsin future is discounted by γt. The agent’s goal is tolearn a policy π that specifies an action to performat each state s, so that the expected discounted, cu-mulative future reward starting from s is maximized.In model-based RL, the optimal policy is estimatedfrom the transition model T and the reward model R.In this paper we concentrate on learning the transi-tion model, with a known reward model. The rewardmodel, however, can be learned in a similar way.

Online Feature Selection for Model-based Reinforcement Learning

In a factored MDP, each state is represented by avector of n state-attributes. The transition func-tion for the factored states is commonly expressedusing dynamic Bayesian networks (DBNs) in whichT (s, a, s′) =

∏ni=1 P (s′i|Paai (s), a), where Paai indi-

cates a subset of state-attributes in s called the parentsof s′i (Fig.1a). Learning T requires learning the subsetsPaai and the parameters for conditional distributions,or the DBN local structures in other words.

Learning DBN structures of the transition functiononline, i.e., while the agent is interacting with theenvironment, however, is computationally prohibitivein most domains. On the other hand, recent stud-ies (Xiao, 2009; Yang et al., 2010) have shown encour-aging results in learning the structure of logistic regres-sion models, which can effectively serve as local struc-tures in the DBNs even in high dimensional spaces.While these regression models cannot fully capture theconditional distributions, their expressive power canbe improved by augmenting low dimensional state rep-resentation with non-linear features of the state vec-tors. We introduce an online sparse multinomial logis-tic regression method that supports efficient learningof the structured representation of the transition func-tion.

2.1. CMDP - a factored MDP withfeature-variables and action effects

We present a variant of the factored MDP that de-fines a “compact but comprehensive” factorization ofthe transition function and supports efficient learn-ing of the relevant features. We consider two majorapproaches to modeling world dynamics: predictingchanges and differentiating features.

First, we predict the relative changes of states insteadof directly specifying the next states in a transition.Mediating state changes via action effects is a commonstrategy in situation calculus (McCarthy, 1963). Sincethe number of relative changes or action effects is usu-ally much smaller than the size of the state space, thecorresponding prediction task should be easier. Thelearning problem can then be expressed as a multi-class classification task of predicting the action effects.

Second, we differentiate the roles of attributes or fea-tures that characterize a state. In a regular factoredMDP, the state-attributes or features serve to both de-fine the state space and capture information about thetransition model. For example, two state-attributes,the (x, y)-coordinates uniquely identify a state andcompactly factorize the state space in a grid-world.A policy can be learned on this factored space. Thetransition dynamics or action effects, however, may

depend on other features of the state, such as the sur-face material at the location (state). Such features areoften carefully included in the state representations.While essential in formulating the transition or rewardmodels, these features may complicate the planning orlearning processes by increasing the size and complex-ity of the state space.

We separate the state identifying state-attributes fromthe “merely” informative state-features in our repre-sentation. This way, we can apply an efficient featureselection method on a large number of state featuresto capture the transition dynamics, while maintaininga compact state space.

More formally, a situation Calculus MDP (CMDP)is defined by a tuple (S, f,A, T,E,R, γ), whereS,A, T,R, γ have the same meaning as in regular MDP.S = 〈S1, S2, .., Sn〉 is the state space implicitly repre-sented by vectors of n state-attributes. The functionf : S → Rm extracts m state-features from each state.E is an action effect variable such that the transitionfunction can be factored as

T (s, a, s′) = P (s′ | s, a) =

n∏i=1

P (s′i | s, f(s), a)

=

n∏i=1

∑e∈E

P (s′i | e, s)P (e | s, f(s), a).

Figure 1b shows an example of this decomposition.The agent uses the feature function f to identify therelevant features, and then use both state attributesand features to predict the action effects. We also as-sume that the effect e and current state s determinethe next state s′, thus P (s′|e, s) is either 0 or 1. Thisdefines the semantic meaning of the effect which isassumed to be known by the agent. The remainingtask is to learn the P (e|s, a) = P (e|x(s), a), wherex(s) = (s, f(s)), which is a classification problem; wesolve this problem using multinomial logistic regres-sion methods.

2.2. Online multinomial logistic regressionwith group lasso

We introduce a regularized online multinomial regres-sion method with group lasso that allows us to learna probabilistic multi-class classifier with online featureselection. We also show that the structure and param-eters of the learnt classifier are likely to converge tothose of the optimal classifier.

2.2.1. Multinomial logistic regression

Multinomial logistic regression is a simple yet effec-tive classification method. Assuming K classes of d-

Online Feature Selection for Model-based Reinforcement Learning

S3

S2

S1

S4

S′3

S′2

S′1

S′4

(a)

E

S2

S1

S′2

S′1

f2(.)

f1(.)

f3(.)

(b)

Figure 1. a.) Standard DBN. b.) Our customized DBNfor CMDP.

dimensional vectors x ∈ Rd, we represent each classk with a d-dimensional prototype vector Wk. Classifi-cation of an input vector x is based on how “similar”it is to the prototype vectors. Similarity is measuredwith inner product 〈Wk, x〉 =

∑di=1Wkixi, where xi

denotes feature i. The log probability of a class isdefined by logP (y = k|x;Wk) ∝ 〈Wk, x〉. The param-eter vectors of the model form the rows of a matrixW = (W1, ...,WK)T .

Let lt(Wt) = − logP (yt|xt;W t) denote the item-wise

log-loss of a model with coefficient matrix W t predict-ing a data point (yt, xt) observed at time t. A typicalobjective of an online learning system is to minimizethe total loss by updating its W t over time. However,the resulting model will often be very complicated andover-fitting. To achieve a parsimonious model, we ex-press our a priori belief that most features are irrel-evant or superfluous by introducing a regularizationterm Ψ(W ) = λ

∑di

√K||W·i||2, where ||W·i||2 denotes

the 2-norm of the ith column of W , and λ is a posi-tive constant. This regularization is similar to that ofgroup lasso. It communicates the idea that it is likelythat a whole column of W has zero values (especially,for large λ). A column of all zeros suggests that thecorresponding feature is not necessary for classifica-tion.

The objective function can now be written as

L(T ) =

T∑t=1

lt(Wt) + Ψ(W t)

=

T∑t=1

− loge

⟨W t

yt ,xt⟩

∑k e〈W t

k,xt〉 + λ

d∑i

√K||W t

·i||2,

where W t is the coefficient matrix learned using t− 1previously observed data items. The quality of a se-quence of parameter matrices W t, t ∈ (1, . . . , T ) withrespect to a fixed parameter matrix W can be mea-

sured by the amount of extra loss, or regret

RT (W ) = L(T )− LW (T )

=

T∑t=1

(lt(Wt) + Ψ(W t))−

T∑t=1

(lt(W ) + Ψ(W ))

We want to learn a series of parameters W t to achievesmall regret with respect to a good model W that hasa small loss LW (T ).

2.2.2. Online learning for regularizedmultinomial logistic regression

We introduce an update function, mDAGL-update(Algorithm 1) to extend the efficient dual averagingmethod (Xiao, 2009) for solving lasso and grouplasso (Yang et al., 2010) logistic regression on binaryclassification to the multi-class case.

Let h(W ) be a strongly convex function with modulus1; W 0 = arg minW h(W ), and let W t=1 be initializedto W 0. Let Gtki be the partial derivatives of functionlt(W ) with respect to Wki at W t (Gtki = ∂lt

∂Wki(W t)).

We define Gt to be a matrix of average partial deriva-tives, i.e., Gtki = 1

t

∑tτ=1G

τki, where

Gτki = −xτi (I(yτ = k)− P (k|xτ ;W τ )). (1)

For any data observed at time t, we update the coeffi-cient matrix via

W t+1 = arg minW

(⟨Gt,W

⟩+ Ψ(W ) +

βtth(W )

),

(2)where βt is a non-negative, non-decreasing constantsequence, and 〈·, ·〉 denotes an inner product betweentwo matrices;

⟨Gt,W

⟩=∑k,i G

tkiWki.

Theorem 1 (Update Rule) Given h(W ) =12 ||W ||

22, a K × d average gradient matrix Gt, and a

regularization parameter λ > 0, the optimal solutionof (2) is achieved column-wise as follows

W t+1·i =

{−→0 if ||Gt·i||2 ≤ λ

√K,

tβt

(λ√K

||Gt·i||2− 1)Gt·i otherwise.

(3)

This rule dictates that when the length of the aver-age gradient matrix column is small enough, the cor-responding parameter column should be truncated tozero. This amounts to feature selection.

The following regret analysis confirms that the solu-tion will converge and that the average maximal regretasymptotically approaches zero with rate O( 1√

t).

Online Feature Selection for Model-based Reinforcement Learning

Algorithm 1 The mDAGL update

Input: t, yt, xt,W t, Gt−1, λ, αGt ← use equation 1 with (yt, xt),W t

Gt ← t−1t G

t−1 + 1tG

t

W t+1 ← use equation 3 with Gt, βt = α√t, λ

return (W t+1, Gt)

Theorem 2 (Regret Bound) Let the sequence of{W t}t≥1 be generated by the update rule (3), and as-sume that there exists a constant G such that ||Gt||22 ≤G2,∀t ≥ 1. If we choose βt = α

√t where α > 0, then

for any t ≥ 1 and for any W that satisfies h(W ) ≤ D2

where D is a constant, the average regret is boundedas

Rt(W )

t≤ ∆√

t, t = 1, 2, 3.., (4)

where ∆ =(αD2 + G2

α

).

[Proof Sketch ] The item-wise loss function lt(W )of multinomial logistic regression is convex, thus thetechniques used for binary case (Xiao, 2009) can beapplied for multinomial case as well.

Since the average regret goes asymptotically to zero,it may look very feasible that the sequence (W t) alsoconverges to some optimal W ∗. However, the regretanalysis is valid for any sequence of data, and with-out additional assumptions about the data generatingprocess there may not be any asymptotically optimalclassifier W ∗, thus convergence is not meaningful. Tostudy convergence, we assume the data is to be sam-pled independently from some joint distribution p fordata vector (y,x). In this case we try to find a W thatminimizes the expected loss Ep[l(W )] + Ψ(W ). Nowassuming that the optimal solution W ∗ is sparse, andsome other technical assumptions, it is indeed possibleto show that

P (||W t −W ∗||2 > ε) <

[ε−1(ε−1 + r−1) +

2

c∆

]t−

14 ,

(5)where r and c are constants (see Lemma 13 in (Lee &Wright, 2012) for the result and its assumptions).

2.3. Model-based RL with feature selection

Our main task is to turn transition model learninginto the learning of conditional distributions P (E |s, f(s), a) using multinomial logistic regression forwhich attention to relevant features can be efficientlyimplemented online via mDAGL.

The key steps of our method, called loreRL (RL withregularized logistic regression), are presented in Al-

Algorithm 2 The loreRL algorithm

Input: mDAGL regularization parameters λ, α,CMDP variables S, f,A,E,R, γ, exploration ε.Let W = (W1,W2, . . . ,W|A|) = (W 0,W 0, . . . ,W 0)

Let G = (G1, G2, . . . , G|A|) = (~0,~0, . . . ,~0)s0 ← random initial statefor t = 1, 2, 3, ... doπ ← Solve MDP using transition model T (W )a← π(st, ε) #ε-greedy action selectionTake action a yielding effect e, next state st+1

(Wa, Ga)← mDAGL(t, e, x(st),Wa, Ga, λ, α)end for

gorithm 2. Inputs to loreRL are the CMDP compo-nents (except the transition function), regularizationparameters λ and α of mDAGL algorithm, and the εthat determines the probability of taking a random ac-tion. We first initialize logistic regression parametersWa and the average gradient matrices Ga for each ac-tion a ∈ A. We also randomly select a starting states0.

At each time step, a random action a is chosen with asmall probability ε, but otherwise we calculate the op-timal policy π for an MDP with the transition modelT (W ) is based on the current effect predictors. Whilewe have used value iteration (like in Rmax) for findingthe optimal policy, any other model-based RL tech-nique can be used as well. We do not focus on theplanning part of RL here, but Dyna-Q or PrioritizedSweeping can be deployed for a more scalable algo-rithm. After performing an action a in state st andobserving its effect e, the experience (e, st, f(st)) willbe presented to the mDAGL algorithm that updatesthe parameter matrix Wa and the gradient matrix Ga.

As we just do ε-greedy random sampling, it is im-possible to guarantee PAC convergence to an optimalpolicy. Assuming that observed data is i.i.d, we canprove that difference in optimal value functions of twoCMDPs with different logistic regression based tran-sition functions is bounded by the difference in theirparameters. This leads to a corollary for convergenceto near optimal policy.

Theorem 3 (Difference in Value Function)Let M1 = (S, f,A, T (WM1), E,R, γ) andM2 = (S, f,A, T (WM2), E,R, γ) be two CMDPswith optimal policies π1 and π2 respectively. Letus denote by VMπ the value function for policy π inCMDP M . Let

ε1 = 2

√max

a∈A,e∈E||W (a),M1

e −W (a),M2e ||1 sup

s||x(s)||1,

Online Feature Selection for Model-based Reinforcement Learning

then maxs∈S(VM2π1

(s)− VM2π2

(s))≤ 2γVmaxε1

1−γ ,

where W(a),M1e and W

(a),M2e refer to the vector of co-

efficients corresponding to class E = e under action ain model M1 and M2 respectively, || · ||1 is the 1-normof vector, and Vmax is the maximum value of any statefor any policy in either of the CMDPs.

By taking M2 to be an CMDP based on the optimalW ∗ and M1 an estimated CMDP based on mDAGL,the vanishing bound given in equation (5) can be trans-lated into a vanishing bound for value difference ofpolicies. In case the true transition model is repre-sentable by a sparse W ∗, we would most probably con-verge to a near optimal policy.

When we cannot express the true transition dynamicsas logistic regression based on available state features,it is hard to give guarantees of performance. However,we can still have some confidence in doing well. The lo-gistic regression model P ∗l closest (in Kullback-Leiblerdistance) to the true model Ptrue (possibly not a logis-tic regression model) is the one1 that has the smallestexpected log-loss. While our optimality criterion is theexpected regularized log-loss, we expect the regular-ized log-loss optimal model P ∗Ψ to be close to P ∗l thusalmost as close to Ptrue as we can get. This relativelysmall KL-distance can be converted to relatively smalldistances in actual transition probabilities, which canthen further be converted to a relatively small boundon value differences by the same arguments used inproving Theorem 3. Therefore, since our model wouldvery likely converge close to P ∗Ψ, we can expect to doalmost as well as P ∗Ψ.

3. Related work

DBN has been a popular choice for factoring and ap-proximating transition models. In DBN learning fea-ture selection is equivalent to picking the parents ofthe state variables from the previous time slice. Re-cent studies have led to improvements in sample com-plexity for learning optimal policy. Those studies as-sume maximum number of possible parents for a node(Strehl et al., 2007), (Diuk et al., 2009), or knowledgeof a planning horizon that satisfies certain conditions(Chakraborty & Stone, 2011). However, the improve-ments in sample complexity are achieved at the ex-pense of actual computational complexity since thesemethods have to search through a large number of par-ent sets. Hence, these methods appear feasible only in

1Such model may not always exist since the parameterset is open. However, for our argument, any model withalmost infimum distance to the true model will do.

manually designed, low-dimensional state-spaces.

Instead of searching for an optimal model with a mini-mal number of samples at almost any cost, our methodattempts to save costs from early on, and graduallyimprove the model acknowledging that the true modelmay actually be unattainable. In this spirit the struc-ture learning study by Degris et al. (Degris et al., 2006)resembles our work, but they do not address onlinelearning with large feature sets. Ross et al. (Ross &Pineau, 2008) have used Markov chain Monte Carloto sample from all possible DBN structures. However,the Markov Chain used has a very long burn-in periodand slow mixing time, making sampling computation-ally prohibitive in large problems.

Kroon and Whiteson (2009) present a feature selec-tion method for learning state values. This method,however, assumes that the DBN structures of transi-tion and reward models are known. Our work, on theother hand, does not make such assumptions.

Leffler et al. (2007) also suggest to predict relativechanges in states, which corresponds to the action ef-fects in our formulation. However, they manually se-lect important features to aggregate information fromsimilar states for action effect predicting. Our workfocuses on learning those features automatically. Hes-ter and Stone (2009; 2012) later employ Quinlan’sC4.5 (Quinlan, 1993) to learn a decision tree for pre-dicting relative changes of every state variable. Thisworks better than the method by Degris et al. Despiteadapting C4.5 for online learning, the method is stillvery slow as a costly tree induction procedure has tobe repeated many times in a large feature space. In ad-dition, all the data needs to be stored for the purpose,which is undesirable in some applications.

Strehl and Littman (2007), and Walsh et al. (2009)have also proposed an online linear regression methodwith L2-regularization to approximate the transitionmodel in continuous MDP. L2, however, does not im-plement feature selection.

4. Experiments

We present empirical evaluation of loreRL on both sim-ulated and real robotic domains. The experiments aimto demonstrate that loreRL can a) generalize and ap-proximate the transition model to achieve fast con-vergence to near optimal policy, and b) with featureselection, perform well in complex, feature rich envi-ronments. We also want to see if theoretical promisesderived under assumption of i.i.d sampling can be real-ized in practice. We compare accumulated rewards ofloreRL with factored Rmax (fRmax), in which the net-

Online Feature Selection for Model-based Reinforcement Learning

work structures of transition models are known (Strehlet al., 2007), and with factored ε-greedy (fEpsG), inwhich the optimistic Rmax exploration of fRmax isreplaced by ε-greedy strategy. We also compare ourmethod with RL-DT (Hester & Stone, 2009) and LSE-Rmax (Chakraborty & Stone, 2011), which are thestate of the art model-based RL algorithms for learn-ing transition models.

4.1. Grid-world domain

In this domain, the agent tries to reach its goal in thegrid-world world consuming as little energy as possi-ble. Each cell in the world has one of five surface ma-terials: sand, soil, water, brick, and fire; there may bewalls between cells. Surface and walls are features thatdetermine the stochastic dynamics of the world. In ad-dition, to test the variable selection aspect, we attachhundreds of random binary features to the environ-ment; it has to learn to focus on the relevant featuresto quickly achieve its goal. The agent can perform fouractions (move up, down, left, right), which will lead itto one of the four states around it or leave it to its cur-rent state. Effects of the actions are captured in fiveoutcomes (moved up, left, down, right, did not move).The states are defined by the (x,y)-coordinates of theagent. To perform an action, agent will spend 0.01units of energy. It loses 1 unit if falling into a stateof fire, but gains 1 unit when successfully reaching anexit door. A task ends when agent reaches a terminalstate, i.e., any exit door or state with fire.

We generated the environment transition models fromfour random multinomial logistic distributions (one foreach action); every different combination of cell sur-faces and walls around the cell will lead to differenttransition dynamics at the cell. The probability of go-ing through a wall is rounded to zero and the freedprobability mass is evenly distributed to other effects.The agent’s starting position is randomly picked ineach episode.

We ran the experiments with loreRL having α = 1.5,λ = 0.05, γ = 0.95, exploration ε = 0.05, parameterm = 10 for fRmax, m = 5 for Rmax (m = 5 is smallfor Rmax, but increasing it did not yield better result),fixed m = 10, σ = 0.99 for LSE-Rmax (similar to theauthor’s report). All results are averaged over 20 runs,and we report the 95% confidence intervals.

Generalization and convergence. We first showthat when the feature space is small, loreRL performsas efficiently as the state of the art methods. RL-DTemploys a decision tree to generalize transition dynam-ics knowledge over states, but it is implemented with ε-greedy exploration strategy. LSE-Rmax appears to be

the best structure learning in ergodic factored MDPs(Chakraborty & Stone, 2011). fRmax and fEpsG havecorrect DBN structures provided by an oracle. All themethods are implemented with our customized DBNto utilize domain knowledge. Rmax is included as areference to show the effect of knowledge generaliza-tion.

As seen in Figure 2a, loreRL can approximate theworld dynamics using samples in all the states, thus itconverges as fast as fEpsG, and RL-DT to near optimalpolicy. Although fRmax is provided with the correctDBN structure, its accumulated reward is lower dueto aggressive exploration to find the optimal model.After exploration the policy is guaranteed to be nearoptimal, but it may still take a long time (or forever)to catch up with loreRL. While LSE-Rmax follows theRmax scheme, it starts with a simple model and ex-plores a bit less aggressively than fRmax, gaining someadvantage in early episodes. However, LSE-Rmax ap-pears to require much more data to choose a morecomplex model. Its accumulated reward drops belowfRmax after 150 episodes, and the angle of the curvesuggests that its DBN structure is still not correct. Wedid not run LSE-Rmax for more episodes, as the algo-rithm has very high computational demand (Table 1).

When the feature set has many irrelevant features(Figure 2b), loreRL is able to learn the relevant onesand still gain nearly as high accumulated reward asfEpsG which has relevant features provided by oracle.Also loreRL’s running time is not much longer thanfRmax’s or fEpsG’s (Table 1). Other methods are tooslow to be run in this high-dimensional environment.

These results also suggest that with ε-greedy explo-ration and random restarts, near optimal policy canbe found even without i.i.d data sampling.

Table 1. Average running time per episode in 800 episodeswhen acting in an environment with 210 features. (SlowRL-DT, LSE-Rmax could only be run with 10 features.)Run on Intel Xeon CPU 2.13GHz, 32GB RAM.

Algorithm fRmax fEpsG RL-DT LSE-R. bloreRL loreRLTime (sec.) 0.26 0.25 9.09 67.53 4.3 0.55

Feature selection. To model real-life situations, thefeature space is usually exponentially large. The abil-ity to focus only on the (most) relevant features isrequired to achieve effective learning. To understandthe role of feature selection, we focus on comparingloreRL with a bloreRL that is based on multinomiallogistic regression without feature selection (withoutthe regularization term). fEpsG and fRmax are baselines.

Online Feature Selection for Model-based Reinforcement Learning

-600

-400

-200

0

200

400

0 200 400 600 800

Acc

um

ula

ted R

ewar

d

No. of episodes

fRmaxfEpsG

RL-DTLSE-Rmax

loreRLRmax

(a) CMDP with 10 features.

-600

-400

-200

0

200

400

0 200 400 600 800

No. of episodes

fRmaxfEpsG

bloreRLloreRLRmax

(b) Extra 200 irrelevant features.

-200

-100

0

100

200

300

400

0 50 100 150 200 250 300 350 400 450

No. of irrelevant features

bloreRLloreRLfRmaxfEpsG

(c) Rewards after 800 episodes.

Figure 2. Accumulated rewards in a 900 state CMDP for various model-based RL-methods.

Figure 2b shows the accumulated reward when the en-vironment has 200 irrelevant binary features. As seen,loreRL is still able to converge fast to optimal pol-icy, and outperforms fRmax and bloreRL. Figure 2cshows performances of loreRL and bloreRL after 800episodes as a function of the number of irrelevant fea-tures. Only minimally affected by the actual numberof irrelevant features, loreRL can quickly select the rel-evant features and outperform bloreRL. loreRL doesnot lose much to fEpsG either. While fRmax may findan optimal policy before loreRL due to agressive ex-ploration, its accumulated reward is still lower thanloreRL’s. In our experiments, we also observed thatloreRL, that selects a small set of features, is muchfaster than bloreRL (Table 1).

4.2. Robotics domain

Our second experiment is conducted in a real environ-ment where we cannot expect the effect of the actionsto follow logistic regression model. The domain (Fig-ure 3) consists of a 5 × 5-feet environment made ofvarious materials such as beans, soil, hay, leaves, andcarpet that cause agent’s actions, move-forward, turn-left, and turn-right, to have different effects at differentlocations. The environment also contains a blue tar-get ball, and there are marks painted in green, blue,and red colors on cardboard surface. The three wheelrobot was built with LEGO Mindstorms NXT kit anda camera was installed above the area so that the robotcan fully observe the environment.

To learn the transition function for the robot, we dis-cretized the environment into a state space of 8 × 8(x,y)-locations and 8 different orientations of a robot,which yields a state space of 512 states. The actionsmay change robot’s relative location in four differentways and orientation in five different ways resultingin total of 20 different effects. However, in differentstates the actions’ tendency to produce effects maybe different due to the differences in surface mate-

rials. To capture this variation, the agent describeseach state with a long vector of binary features. The“green” binary indicator fGi (s) of a state s is set to 1iff there is a green mark that is further than i units butcloser than i+ 1 units from the xy-center of the states (i ∈ {0, . . . , 99}). Similar features are defined forblue marks and to the blue target ball yielding 300 bi-nary features. Eight indicators for different robot ori-entations are also included in the feature-base togetherwith four intentionally redundant “there is/is-not agreen/blue mark in the state grid”-bits. All togetherthese yield 312 binary features per state. The intu-ition behind these features is that they serve as proxiesto surface materials, slopes on the surfaces, obstacles,etc., which are likely to be important factors deter-mining the dynamics in the environments, but whichthe robots sensors cannot capture. Although only fewamong these 312 features are important for modelingrobots actions, the robot does not know those criticalfeatures. The robot has to learn to select them basedon feedback while interacting with the environment.

The robot’s task is to travel in the environment froma random start and reach the blue ball, which will earnit a reward of 2 points. The robot will receive −1point if it falls out of the area or into the death placesmarked with orange rectangles, and −0.05 points foran action at any other states. An episode ends if therobot reaches a terminal state, or gets stuck for fourconsecutive actions.

The robot battery did not allow us to compare ouralgorithm with the slow RL-DT and LSE-Rmax; wecould only compare with the fine-tuned fRmax, fEpsG,and man-loreRL algorithms in which we manuallyselected important features and specified the DBN-structures for the transition models. man-loreRL isbased on multinomial logistic regression models with12 manually selected features. We ran the experimentswith loreRL and man-loreRL having α = 0.5, λ = 0.05,γ = 0.95, exploration ε = 0.05, parameter m = 10 for

Online Feature Selection for Model-based Reinforcement Learning

fRmax. All results are averaged over 10 runs, and wereport the 95% confidence intervals.

Figure 3. A real environment.

-20

0

20

40

60

0 10 20 30 40 50

Acc

um

ula

ted R

ewar

d

No. of episodes

fRmaxfEpsG

loreRLman-loreRL

Figure 4. Accumulated rewards of various methods.

In Figure 4, loreRL appears to quickly capture theenvironment dynamics and outperform other meth-ods. Even with manually selected features, fRmax andfEpsG require more exploration to learn the dynamics.man-loreRL gains rewards a bit faster, but in the endit slightly loses to loreRL possibly due to the (unfore-seen) insufficiency of the manually selected features.Table 2 further shows that loreRL is fast. Its aver-age running time per episode with 312-features is onlyslightly slower than with 12 manually selected features.

Table 2. Average running time per episode in 50 episodes.Run on Intel Centrino Duo T2400 (1.83GHz), 1GB RAM.

Algorithm fRmax fEpsG man-loreRL loreRLTime (sec.) 134.37 127.69 93.54 108.10

5. Conclusions

We have demonstrated how online multinomial logisticregression with group lasso can be used to quickly ob-tain a parsimonious transition model in model basedRL. The method leads to fast learning since a singletransition model can be learnt using samples from allthe states with a small set of features.

The efficiency is gained, however, at the expense oflosing generality. Not all transition functions can beaccurately represented as predicting action effects us-ing state features via logistic regression. Nevertheless,we believe that this compromise between scalabilityand generality is often a useful one. The generalityproblem may also be alleviated by introducing non-linear features that are combinations of the originalones.

Other generalizations such as stochastic features andvector valued effects are also possible but are left forfuture work. The proposed framework also opens anopportunity for knowledge transfer. While differentenvironments often have different states, the effects ofactions are likely to persist from environment to en-vironment. mDAGL would allow an agent to learntransferrable sparse action models. For instance, thismethod could be incorporated into the TES framework(Nguyen et al., 2012) to implement an intelligent agentthat can learn, accumulate and transfer knowledge au-tomatically between environments.

Acknowledgments

We thank PhD. Haiqin Yang for helpful comments onour sparse multinomial logistic regression. This re-search is supported by Academic Research Grants:MOE2010-T2-2-071 and T1 251RES1005 from theMinistry of Education in Singapore.

References

Brafman, Ronen I. and Tennenholtz, Moshe. R-max- a general polynomial time algorithm for near-optimal reinforcement learning. In Journal of Ma-chine Learning Research, 3:213–231, 2003.

Chakraborty, Doran and Stone, Peter. Structure learn-ing in ergodic factored MDPs without knowledge ofthe transition function’s in-degree. In Proceedings ofthe International Conference on Machine Learning,ICML’11, pp. 737–744, 2011.

Degris, Thomas, Sigaud, Olivier, and Wuillemin,Pierre-Henri. Learning the structure of factoredMarkov decision processes in reinforcement learningproblems. In Proceedings of the International Con-ference on Machine Learning, ICML’06, pp. 257–264, 2006.

Diuk, Carlos, Li, Lihong, and Leffler, Bethany R. Theadaptive k-meteorologists problem and its applica-tion to structure learning and feature selection in re-inforcement learning. In Proceedings of the Interna-

Online Feature Selection for Model-based Reinforcement Learning

tional Conference on Machine Learning, ICML’09,pp. 249–256, 2009.

Hester, Todd and Stone, Peter. Generalized modellearning for reinforcement learning in factored do-mains. In Proceedings of International Conferenceon Autonomous Agents and Multiagent Systems,volume 3 of AAMAS’09, pp. 717–724, 2009.

Hester, Todd and Stone, Peter. Texplore: real-timesample-efficient reinforcement learning for robots.Machine Learning, pp. 1–45, 2012.

Kroon, Mark and Whiteson, Shimon. Automatic fea-ture selection for model-based reinforcement learn-ing in factored MDPs. In Proceedings of the Inter-national Conference on Machine Learning and Ap-plications, ICMLA’09, 2009.

Lee, S. and Wright, S. Manifold identification of dualaveraging methods for regularized stochastic onlinelearning. In Journal of Machine Learning Research,13:1705–1744, 2012.

Leffler, Bethany R., Littman, Michael L., and Ed-munds, Timothy. Efficient reinforcement learningwith relocatable action models. In Proceedings ofthe National Conference on Artificial Intelligence,AAAI’07, pp. 572–577, 2007.

McCarthy, John. Situations, actions, and causal laws.Technical Report Memo 2, Stanford Artificial Intel-ligence Project, Stanford University, 1963.

Nguyen, Trung T., Silander, Tomi, and Leong,Tze Yun. Transferring expectations in model-basedreinforcement learning. In Proceedings of the Ad-vances in Neural Information Processing Systems,NIPS’12, 2012.

Quinlan, J. R. C4.5: Programs for Machine Learning.Morgan Kaufmann, 1993. ISBN 1-55860-238-0.

Ross, Stephane and Pineau, Joelle. Model-basedbayesian reinforcement learning in large structureddomains. In Proceedings of the Conference on Un-certainty in Artificial Intelligence, UAI’08, pp. 476–483, 2008.

Strehl, Er L. and Littman, Michael L. Online linearregression and its application to model-based rein-forcement learning. In Proceedings of the Advancesin Neural Information Processing Systems, NIPS’07,pp. 737–744, 2007.

Strehl, Er L., Diuk, Carlos, and Littman, Michael L.Efficient structure learning in factored-state MDPs.In Proceedings of the National Conference on Arti-ficial Intelligence, AAAI’07, 2007.

Walsh, Thomas J., Szita, Istvan, Diuk, Carlos,and Littman, Michael L. Exploring compactreinforcement-learning representations with linearregression. In Proceedings of the Conference on Un-certainty in Artificial Intelligence, UAI’09, pp. 591–598, 2009.

Xiao, Lin. Dual averaging methods for regularizedstochastic learning and online optimization. In Pro-ceedings of the Advances in Neural Information Pro-cessing Systems, NIPS’09, 2009.

Yang, Haiqin, Xu, Zenglin, King, Irwin, and Lyu,Michael R. Online learning for group lasso. In Pro-ceedings of the International Conference on MachineLearning, ICML’10, pp. 1191–1198, 2010.