JOIMwww.joim.com

Journal Of Investment Management, Vol. 18, No. 2, (2020), pp. 1–20

© JOIM 2020

DYNAMIC GOALS-BASED WEALTH MANAGEMENTUSING REINFORCEMENT LEARNING

Sanjiv R. Dasa and Subir Varmaa

We present a reinforcement learning (RL) algorithm to solve for a dynamically optimalgoal-based portfolio. The solution converges to that obtained from dynamic programming.Our approach is model-free and generates a solution that is based on forward simulation,whereas dynamic programming depends on backward recursion. This paper presents abrief overview of the various types of RL. Our example application illustrates how RL maybe applied to problems with path-dependency and very large state spaces, which are oftenencountered in finance.

1 Introduction

Reinforcement learning (RL) has seen a resur-gence of interest as the methodology has beencombined with deep learning neural networks.Advances in hardware and software have enabledRL in achieving newsworthy successes, such aslearning to surpass human-level performance invideo games (Mnih et al., 2013) and beatingthe world Go champion (Silver et al., 2017).RL algorithms are particularly good at learningfrom experience and at optimizing the “explore–exploit” tradeoff inherent in dynamic optimiza-tion problems. This is why they can be fine-tuned

aSanta Clara University

to play dynamic games with superhuman levelsof performance. The capacity of RL algorithms tolearn from repeated trial episodes of games can beaccelerated to a degree mankind cannot replicate.It is just not possible for anyone to play a milliongames over a weekend, whereas a machine can.

The dynamic optimization of portfolio wealthover long horizons is similar to optimal gameplay. Portfolio managers choose actions—that is,asset allocations, and hope to respond to marketmovements in an optimal manner with a view tomaximizing long-run expected rewards. In thispaper, we show how RL may be employed tosolve a particular class of portfolio “game” knownas goals-based wealth management (GBWM).GBWM has recently gained widespread accep-tance by practitioners and the use of RL willinform the growth in this paradigm.

Second Quarter 2020 1

2 Sanjiv R. Das and Subir Varma

Since the seminal work of Markowitz (1952),there has been a vast literature on portfolio opti-mization, broadly defined as allocating moneyto a collection of assets (portfolio) with an opti-mal tradeoff between risk and return. In prac-tice, return is defined as the weighted averagemean return of the portfolio and risk is usuallythe standard deviation of this weighted return.Markowitz’s early work detailed a static (one-period) optimization problem that forms the ker-nel for many dynamic optimization problems,where a statically optimal portfolio is chosen eachperiod in a multiperiod model in order to maxi-mize a reward function at horizon T . The rewardfunction may be (1) based on a utility function,or (2) on whether the final value of the portfo-lio exceeds a desired threshold. The latter typeof reward function underlies the broad class ofgoals-based wealth management (GBWM) mod-els, see Chhabra (2005), Brunel (2015), and Daset al. (2018).

In GBWM, we seek to maximize the probabilitythat an investor’s portfolio value W will achievea desired level of wealth H—that is, W(T ) ≥ H

at the horizon T for the goal. Starting from timet = 0, with wealth W(0), every year, we willchoose a portfolio that has a specific mean return(μ) and risk (denoted by the standard deviationof return σ) from a set of acceptable portfolios, soas to maximize the chance of reaching or exceed-ing threshold H . We may think of this as playinga video game where we see the portfolio movein random fashion, but we can modulate the ran-domness by choosing a (μ, σ) pair at every movein time. This game may be solved in two ways:(i) At time T , we consider all possible values ofwealth W(T ) and assign high values to the out-comes where we exceed threshold H and lowvalues to outcomes below H . We then work back-wards from all these possible values of wealthto possible preceding values and work out whichmove (μ, σ) offers the highest possible expected

outcome. This approach is widely used and isclassic dynamic programming, see Das et al.(2019). It is known in the reinforcement learning(RL) literature as solving the “planning prob-lem.” We provide formal details on this brieflyin Section 5.

Dynamic programming (DP) via backward recur-sion is oftentimes hard to implement becausebackward recursion is computationally costly,usually because (1) the number of states in thegame is too large, or (2) the transition proba-bilities from one state to another are unknownand need to be estimated. Instead, we may resortto forward propagation game simulations, andimprove our game actions by playing repeatedgames and learning from this play as to whichactions are optimal for each state of the game wemay be in. This approach, known as RL, has alsobeen widely studied, see for example, Sutton andBarto (1998).1 The video game analogy for RLhas become popular since Mnih et al. (2013) usedthe approach to beat human-level performance atplaying the Atari video game and many more. Inthis paper, we survey the various kinds of RL andshow how we may solve a multiperiod retirementportfolio problem—that is, optimize GBWM.

This paper proceeds as follows. In Section 2 wereview static mean–variance optimization and setup the dynamic multiperiod goal-based optimiza-tion problem as an example of game playingthat may be solved using RL. In Section 3, wereview how the set of efficient portfolios that com-prise the action space in the model is computed,using the mean–variance solution of Markowitz(1952). Section 4 formulates the dynamic prob-lem in terms of Markov Decision Processes(MDPs). Section 5 describes the various typesof DP and RL algorithms that we may consider.Our taxonomy discusses (i) model-based versusmodel-free RL approaches, (ii) value iterationversus policy iteration as a solution approach, (iii)

Journal Of Investment Management Second Quarter 2020

Dynamic Goals-Based Wealth Management Using Reinforcement Learning 3

on-policy versus off-policy approaches, and (iv)discrete-state space solutions versus continuous-state space solutions that use deep learning neuralnets. Section 6 presents the specific Q-Learningalgorithm we use to solve the dynamic portfo-lio problem. This approach will be model-free,policy iterative, off-policy, and embedded in adiscrete state space. Section 7 reports results ofillustrative numerical experiments, and Section 8offers concluding discussion.

2 Portfolio optimization: Statics anddynamics

In this section, we briefly recap static mean–variance optimization, define GBWM, recastdynamic portfolio optimization as a game, andintroduce RL as a feasible solution approach.

2.1 Mean–variance optimization

The objective of Modern Portfolio Theory is todevelop a diversified portfolio that minimizesrisk—that is, the variance of portfolio return,for a specified level of expected (mean) return.This problem, known as mean–variance port-folio optimization, takes as input a vector ofmean returns M = [M1, . . . , Mn]T and covari-ance matrix

∑of returns of n assets. Once the

investor specifies a required expected return μ forthe portfolio, a set of asset weights w = {wj} =[w1, . . . , wj, . . . , wn]T, 1 ≤ j ≤ n, is chosen tominimize portfolio return variance σ2. The port-folio expected return and standard deviation arefunctions of the inputs—that is,

μ =n∑

j=1

wjMj = wT ·M; σ =√

wT ·∑·w

The asset weights are proportions of the amountinvested in each asset, and these must add upto 1—that is,

∑nj=1 wj = wTO = 1, where

O = [1, 1, . . . , 1]T ∈ Rn. Also, the portfoliomust deliver the expected return μ. For every μ,

there is a corresponding optimal σ, obtained bysolving this portfolio problem for optimal weightsvector w. This collection of (μ, σ) pairs is calledthe “Efficient Frontier” of portfolios.

An investor’s wealth is statically managed bychoosing the best portfolio from this “efficientset” at any point in time to dynamically meether goals. This is the traditional approach to theportfolio optimization problem and it proceeds bychoosing the portfolio that minimizes the overallportfolio risk σ, while achieving a given returnμ. See Das et al. (2018) for the static optimiza-tion problem and Merton (1969, 1971), for thedynamic programming solution to the multiperiodproblem in continuous time.

2.2 Goals-based wealth management

Recent practice is leaning towards an alternativeformulation of the portfolio optimization prob-lem that uses the framework of GBWM. In thisformulation risk is understood as the probabilityof the investors not attaining their goals at theend of a time period, as opposed to the standarddeviation of the portfolios. However, there is amathematical mapping from the original mean–variance problem to the GBWM one, as explainedin Das et al. (2010). Whereas this problem is astatic one, the dynamic version of this GBWMproblem is solved in Das et al. (2019), where DPisused to solve the long-horizon portfolio problem.

DP has been used to solve multiperiod problemsin finance for several decades. The essence of theproblem may be depicted as follows. At any pointin time t, an investor’s retirement account has agiven level of wealth W(t). The investor is inter-ested in picking portfolios every year to reach atarget level of wealth H at time T —that is, shewants that W(T ) ≥ H . Of course, this is not guar-anteed, which means that the goal will only be metwith a certain level of probability. The GBWMoptimization problem is to reach the goal with as

Second Quarter 2020 Journal Of Investment Management

4 Sanjiv R. Das and Subir Varma

high a level of probability as possible—that is,with objective function:

maxw(t), t<T

Pr[W(T ) > H]This entails choosing a portfolio at each t, witha corresponding level of mean return and risk,w(t) = (μ, σ)t , where these portfolios are cho-sen from a select set of “efficient” portfolios,mentioned in Subsection 2.1 and described in thefollowing Section 3. (Be careful not to mix upw ∈ Rn, which is a vector of asset weights, andW(t), which is a scalar level of wealth at period t.)Thus, we solve for an optimal “action” w[W(t), t],a function of the “state” [W(t), t]. Here the statehas two dimensions and the action chooses one ofa collection of possible (μ, σ) pairs, which deter-mines the range of outcomes of the wealth at thenext period, W(t + 1).

2.3 Dynamic portfolio optimization as a game

If you think of portfolio optimization as a gamewith a specific goal (with a corresponding payoffknown as the “reward function”), then DP deliv-ers a strategy telling the investor what risk–returnpair portfolio to pick in every eventuality that maybe encountered along the path of the portfolio,such that it maximizes the probability of reachingand exceeding the pre-specified threshold valueH . The solution approach uses a method knownas “backward recursion” on a grid, which is intu-itive. Create a grid of wealth values W for all timeperiods—that is, [W(t), t] (think of a matrix withtime t on the columns and a range of wealth valuesW on the rows). The reward in column T is either1 (if W(T ) ≥ H) or 0 otherwise, signifying thebinary outcome of either meeting the goal or not.From every wealth grid point at time T−1, we cancompute the probability of reaching all grid pointsat time T . We then pick the portfolio—that is, a(μ, σ) pair, that maximizes the expected rewardvalues at timeT for a single node at timeT−1, andwe do this for all nodes at time T −1. Computing

the expected reward assumes that the transitionprobabilities from state i at time T −1 to state j attime T —that is, Pr[Wj(T ), T |Wj(T−1), T−1],are known (we will describe these probabilitiesin the next section). We have then found theoptimal action—that is, A = (μ, σ) choice, forall nodes at T − 1, and we can also calculatethe expected final reward at each of the T − 1nodes. This expected final reward, determined atall states [W(T−1), T−1], is known as the “valuefunction” at each state. Then, proceed to do thesame for all nodes at T − 2, using the rewardsat all nodes at time T − 1. This gives the opti-mal action and expected terminal reward (valuefunction) for each node at time T − 2. Keep onrecursing backwards until time t = 0. What wethen have is the full solution to the GBWM prob-lem at every node on the grid—that is, in onesingle backward pass. This is an extremely effi-cient problem-solving approach and gives the fullsolution to the problem in one pass. This problemsetup is called the “planning problem” and the DPsolution approach is easily implemented becausethe transition probabilities from one state to thenext are known.

So far, DP via backward recursion has served itspurpose well because the size of the problemshas been small in terms of the number of statevariables and action variables, and the transitionprobabilities are known. The computational com-plexity of the problem depends on the numberof states—that is, the number of grid points wechoose, which is tractable when the state com-prises just wealth W and time t. Complexity alsodepends on the number of possible actions tochoose from as each of these has to be explored.In the GBWM case, this depends on the num-ber of choice portfolios we consider. Backwardrecursion considers all possible scenarios (states)that might be encountered in the portfolio opti-mization game and computes the best action foreach state. It exhaustively enumerates all game

Journal Of Investment Management Second Quarter 2020

Dynamic Goals-Based Wealth Management Using Reinforcement Learning 5

positions and works well when the number ofcases to be explored is limited. Compare thisapproach to building a program to play chess,where the goal is to solve for the best actionfor all possible board configurations, an almostimpossible task even to describe the state spacesuccinctly.

2.4 Reinforcement learning solutions

When the state and action space becomes large,and it becomes difficult to undertake exhaustiveenumeration of the solution, then learning fromexperience through game playing is often a moreefficient solution approach. In situations wherethe model of the environment is unknown—that is, transition probabilities in the large statespace are not available a priori, then explo-ration through game playing is required to obtainan estimate of transition probabilities. Note thatbackward recursion is really an attempt to specifya solution to all possible game scenarios one mayexperience, which is a limit case of learning fromlimited experience through game playing.

In the portfolio optimization game, a player expe-riences a level of wealth W at each point in timeand takes an action A = (μ, σ). A series of suchstate–action (S, A) pairs in sequence, through tohorizon T , is a single game and results in a rewardof 1 or 0. Clearly, we want to increase the likeli-hood of playing (S, A)pairs that lead to rewards of1 and downplay those that lead to 0 rewards. Thatis, we “reinforce” good actions. RL is the processof training our model through repeated play ofportfolio optimization game sequences. We notethat in both DP and RL, actions are a functionof the current state and not preceding states. Thisproperty of the sequence of actions characterizesit as a Markov decision process (MDP), which wewill describe later in this paper.

Through the RL process the agent builds up aset of actions A(W(t), t) to be taken in each state

[W(t), t]. The set of actions is also known as the“policy,” which is a function of the state space.As the agent plays more episodes, the policy isupdated to maximize expected reward. With theright game playing setup, the policy will convergeto the optimal in a stable manner.

Unlike DP, which is solved by backward recur-sion, RL is a forward iteration approach—thatis, we play the game forward and assess the ulti-mate reward. RLdoes not visit each possible state,only a certain number, determined by how manygames, known as “episodes” in the RL pantheon,we choose to play and the behavior of the randomprocess governing the evolution of the system.The hope is that RLis efficient enough to approachthe DP solution with far less computational andalgorithmic complexity. In this paper, we solvethe GBWM dynamic portfolio problem using avariant of RL, known as “Q-Learning” (QL). Wecross check that the solution obtained by DP isalso attained by QL, affirming that RL works asintended for dynamic portfolio optimization.

3 Candidate portfolios

Our problem is restated as follows: Assume thatthe portfolios have to be chosen at fixed inter-vals (h = 1 year) and discrete periods t =0, 1, 2, . . . , T and the amount of wealth at time t isgiven by W(t), 0 ≤ t ≤ T . The threshold return isdenoted as H , and the dynamic GBWM game is tochoose the action A[W(t), t] = (μ, σ)W(t),t drivenby portfolio weights for wealth Wj at time periodt, wj(t), at each time period wj(t), 1 ≤ j ≤ n,0 ≤ t ≤ T−1, such that the probability at the finaltime T of the total wealth exceeding H , given byP(W(T ) > H), is maximized.

Each period, we restrict our portfolios to a dis-crete set of portfolios—that is, (μ, σ) pairs, thatlie along what is known as the “efficient frontier,”described in the previous section as the solutionto a problem where we find a locus of points in

Second Quarter 2020 Journal Of Investment Management

6 Sanjiv R. Das and Subir Varma

Figure 1 The efficient frontier of (σ, μ) pairs, whichare possible portfolio choices available to the investoreach year.

(σ, μ) space (see Figure 1) such that for each μ

we have the portfolio w that minimizes varianceσ2 = wT ∑

w. The mathematics in Das et al.(2018) shows that the equation for the efficientfrontier is as follows:

σ =√

aμ2 + bμ+ c

This curve traces out a hyperbola, as shownin Figure 1. The values a, b, c are given bya = hT

∑h, b = 2gT

∑h, c = gT

∑g,

where the vectors g and h are defined by g =l∑−1

O−k∑−1

M

lm−k2 , h = m∑−1

M−k∑−1

O

lm−k2 , and the

scalars k, l, m are defined by k = MT∑−1

O,l = MT

∑−1M, m = OT

∑−1O. In these equa-

tions M = [M1, . . . , Mn]T is the vector of the n

expected returns of the portfolio assets, O is thevector of n ones, and

∑is the covariance matrix

of the n assets.

In the rest of this paper we restrict the portfo-lio choice at all instants of time 0 ≤ t ≤ T toone of a set of K evenly spaced portfolios thatlie along the Efficient Frontier curve, given by(μ1, σ1), . . . , (μK, σK). We note that the solu-tion here is a one-period solution that deliversthe best locus of portfolios that are inputs intothe action space of the RL algorithm. The opti-mal one-period choice is applied each period in

dynamic manner leading to an optimal dynamicprogramming solution. Note that choosing theportfolio with the best return may not be the opti-mal policy, since higher returns also come withhigher variance, which may cause degradation ofthe objective function.

4 Formulation as a Markov decisionprocesses (MDP)

We now move from the static problem to consid-ering the dynamic portfolio game. At any givenstate—that is, level of wealth W(t), we choose anaction w(t) ≡ A(t) ≡ [μ(t), σ(t)]. The next stepin the game is to generate the next period stateW(t + h), which is stochastic. The time betweentransitions is h. For illustrative purposes, wechoose a stochastic process that transitions W(t)

to W(t + h), and we choose geometric Brownianmotion (GBM), as in Das et al. (2019):

W(t + h)

= W(t)exp

{[μ(t)− 1

2σ2(t)

]h+ σ(t)

√h · Z

}(1)

where Z ∼ N(0, 1). Therefore, randomness isinjected using the standard normal random vari-able Z. GBM is one of the most popular choicesused in financial modeling and therefore, weemploy it here. However, the choice of Z asGaussian is not strict and we may use any otherdistribution without loss of generality.

We are now ready to formulate the portfolio opti-mization problem as an MDP. The MDP state isdefined as (W(t), t), where W(t) is the value ofthe portfolio at time t. For example the portfoliovalue is W(0) at the start of the MDP, and the MDPterminates at t = T when the portfolio value isW(T ). RL is then used to find the optimal policyfor the MDP through playing repeated episodes.A policy is a mapping from the state {W(t), t} to

Journal Of Investment Management Second Quarter 2020

Dynamic Goals-Based Wealth Management Using Reinforcement Learning 7

Figure 2 Evolution of the total portfolio value with time. Choose the portfolios at t = T −1 so as to maximizethe probability of W(T ) being greater than the goal threshold.

action A = {μ, σ}. Each episode will proceed asfollows:

1. Starting with an initial wealth of W(0) at timet = 0, we implement one of the k = 1 . . . K

portfolios, denoted by [μ(0), σ(0)]k. Whichportfolio (action) is chosen will depend on thecurrent policy in place. (At the outset, we mayinitialize the policy to be a guess, possiblyrandom, or some predetermined k.)

2. Using the Geometric Brownian Motion model,the wealth W(t + h) at time t + h is a randomvariable, given by the transition Equation (1).Using this formula, we can sample from therandom variable Z to generate the next wealthvalue W(t + h) at time t + h, conditional onchoosing an action [μ(t), σ(t)]. We then onceagain use our policy to choose the next port-folio [μ(t + h), σ(t + h)]. The sequence ofthese portfolios is an MDP. In our example,we have chosen the transition equation explic-itly to generate the next state, but we are goingto solve the problem as if we do not knowEquation (1).

3. Set t → t + h. If t = T then stop, otherwisego back to step 2.

The system described above evolves in discretetime and action space, but in continuous wealthspace (see Figure 2). For implementing the Q-learning solution later we will restrict the wealthspace to be discrete as well. This approach may begeneralized to continuous spaces by using func-tion approximations based on neural networks.

4.1 State space

Our approach is to define a large range of

[Wmin, Wmax] = [W(0)exp(−3σmax√

T ),

W(0)exp(+3σmax√

T )]at the end of the time horizon T , which willbe an array of final values of W(T ), suitablydiscretized on a grid. Here σmax is taken to bethe highest possible standard deviation of returnacross all candidate portfolios from Section 3.The number of grid points on the wealth gridis taken to be (10T + 1), so if T = 10, thenwe have 101 grid points. The grid points areequally spaced in log-space—that is, equally overthe range [ln(Wmin), ln(Wmax)]. From the ini-tial wealth W(0), we assume that it is possibleto transition to any wealth value on the grid at the

Second Quarter 2020 Journal Of Investment Management

8 Sanjiv R. Das and Subir Varma

end of the period (t = h), though the probabilityof reaching extreme values is exceedingly small.From each wealth value at t = h we assume tran-sition is possible to all the wealth grid values att = 2h, and so on.

Hence, the grid is “fully connected.” Transitionprobabilities from a node i at t to node j at t + h

are described in the next subsection.

4.2 Transition probability

Using Equation (1) that describes the evolution ofW(t), we can write down the following equationfor the transition probability from state Wi(t) attime t to state Wj(t+h) at time t+h (a transitionfrom node i to node j):

Pr[Wj(t + h) |Wi(t)]

= φ

⎡⎣ ln

(Wj(t+h)

Wi(t)

)− (μ− 0.5σ2)h

σ√

h

⎤⎦

where φ(·) is the normal probability densityfunction (pdf) because the Z in Equation (1) isGaussian. In order to fully define the MDP, wealso need to specify the reward function. In thismodel, the reward R is only specified for the finalstates W0(T ), . . . , W10T (T ), and is as follows:R = 1 if Wj(T ) ≥ H , and 0 otherwise. This isakin to a video game reward—that is, you get 1 ifyou win the game and 0 if you lose.

In the case where the investor is allowed to makeinfusions I(t) into the portfolio over time, transi-tion probabilities are adjusted to account for theseadditional cash flows coming into the portfolio.The revised transition probabilities are as follows:

Pr[Wj(t + h) |Wi(t), I(t)]

= φ

⎡⎣ ln

(Wj(t+h)

Wi(t)+I(t)

)− (μ− 0.5σ2)h

σ√

h

⎤⎦

Because the transition probabilities are known,but we solve the problem through forward simu-lation, we are not explicitly using the transitionprobabilities in determining the optimal actions.We only use the transition probabilities to gen-erate the next state with the correct probabilities.The analogy here to video gaming (for this portfo-lio game) is that the game designer has to use sometransition probabilities with which states are gen-erated in the game, but these are not given to thegame player. So, as problem designer, we definethe system and its transition probabilities usingEquation (1) but we do not use these explicitlyto discover the solution, i.e., the optimal policy.Therefore, the RL solution approach we employin this paper is an example of model-free RL. Thedistinction between model-based and model-freeRL is discussed next and we describe a broadtaxonomy of RL approaches.

5 RL taxonomy: Methods for solvingthe MDP

A “policy” π(s) is defined as a mapping fromthe state s to one of the portfolios in the set A

of actions {(μ1, σ1), . . . , (μK, σK)}. The optimalpolicy π∗(s) is the one that maximizes the totalexpected reward, which in this case is the prob-ability of the final value Wi(T ) exceeding thethreshold H . Solving the MDP is the process ofidentifying this optimal policy.

MDP solution methods can be classified into thefollowing categories:

1. Model-based algorithms: These algorithmsassume that the state transition probabilitiesare known. For a given policy π, they are basedon the concept of a State Value Function Vπ(s),s ∈ [W(t), t], and the State–Action ValueFunction Qπ(s, a), a ∈ A. The Value Func-tion Vπ(s) is the expectation of total rewardstarting from state s under policy π, whilethe State–Action Value Function Qπ(s, a) is

Journal Of Investment Management Second Quarter 2020

Dynamic Goals-Based Wealth Management Using Reinforcement Learning 9

the expectation of total reward starting fromstate s and using a as the first action, and pol-icy π thereafter. Similarly, V∗(s) and Q∗(s, a)

are the corresponding value functions that areattained when using the optimal policy π∗. Thevalue function under the optimal policy satis-fies the Bellman Optimality Equation (BOE),see Bellman (1952, 2003) and Bellman andDreyfus (2015), which is a formal statement ofthe backward recursion procedure described inSection 2.3:

V∗(st) = maxat

E[V∗(st+1) | st]

= maxat

E

{maxat+1

E[V∗(st+2) | st+1] | st}

= maxat+h,...,aT−h∈π(s)

E[V∗(sT ) | st](2)

where the last equation follows from the lawof iterated expectations and the Markov prop-erty. This is a simple version of the Bellmanequation because the reward is only receivedat maturity and there are no intermediaterewards. The same equation without the “max-imization” is simply the Bellman ExpectationEquation (BEE) and gives the value of anypolicy, which may not be optimal. A simi-lar equation holds for the State–Action Valuefunction, also known as the Q function (for“quality”):

Q∗(st, at) = maxat,at+h,...,aT−h∈π(s)

E[Q∗(sT ) | st](3)

Once we know either V∗(s) or Q∗(s, a), we canreadily compute the optimal policy. To arriveat these optimal functions, one of the followingalgorithms is used:

• Value Function Iteration.• Policy Function Iteration.

Both of these are iterative algorithms that workby updating value and/or policy functionsthrough episodes of game playing. Value iter-ation iterates on the BOE, whereas policyiteration iterates on the BEE.

RL may be implemented on a discrete or con-tinuous state and action space. If both stateand action spaces are discrete and finite (andof small size), then it is feasible to maintaingrids (tensors of any dimension) for state andaction, and solve for V∗(s) and Q∗(s, a) at eachpoint on the grid. This is known as “tabular”RL. In our GBWM problem, the state spacehas two dimensions W(t) and t, and the actionspace has one (K portfolios), so the tabulargrid will be of dimension three—that is, V∗(s)and Q∗(s, a) will be values on a 3D tensor.To get some quick intuition about theapproach, we define the following componentsof the algorithm.

• State s(t): The current value of variables onwhich the decision (action) is based. In ourexample, this is the level of wealth W(t) andtime t, and is represented by a node on thegrid.• Action a(s(t)): Define the action a as an ele-

ment in the index set {1, 2, . . . , K}, and thepolicy π(s) as a mapping from the state s

to one of the elements in the action set. Inour case, this is the portfolio chosen untilthe next state is realized, at which timeanother action will be taken. The series ofactions is often denoted as a “plan” andhence, learning is analogous to solving a“planning” problem, the result of which isa policy—that is, resulting in a series ofactions (a0, . . . , aT−h).• Reward r(s(t), a(t)): At each state, the agent

may or may not receive a reward for theaction taken. In our example, rewards areonly received at the final horizon T of theproblem.

Second Quarter 2020 Journal Of Investment Management

10 Sanjiv R. Das and Subir Varma

• Transition probability p[s(t+h) | {s(t),a(t)}]: This defines the likelihood of movingto a probabilistic state the following period,conditional on the current state and action.

The value function V(s(t)) is defined over thesame grid. The solution procedure for thisproblem consists of starting from time T andpopulating the value function in the last sectionof the grid—that is, V(s(T )). For our problem,the value is binary—that is, if W(T ) ≥ H ,then V(s(T )) = 1, else it is equal to 0. Oncewe have populated the value function at timeT , we can proceed to populate the value func-tion at time (T − h), using backward iterationbased on the Bellman equation.

V(s(T − h))

= maxa(s(T−h)

E[V(s(T))|s(T − h)]

= maxa(s(T−h))

[m∑

i=0

{p(s(T ) | s(T − h) = i,

a(T − h))} · V(s(T ))

](4)

where an expectation has been taken overvalues V(T ) in all m states in the nextperiod using the transition probability functionp(s(T ) | s(T − h), a(T − h)). This equationembodies the “backward recursion” solutionprocedure, because the same equation may beapplied for all periods from t = T−h to t = 0.This procedure is value iteration and we firstsolved the GBWM problem this way using DPto determine the optimal value function.

In policy iteration, we choose a random pol-icy, and then find the value function of thatpolicy (policy evaluation step). Find a newand better policy based on the previous valuefunction, and so on, until no further improve-ment is possible. During each iteration of the

algorithm, the BEE can be used to computethe value function for the current policy. Herethe policy is explicitly chosen, starting from aninitial functional guess. Standard DP is almostalways amenable to policy iteration, and wehave solved the portfolio problem using RLthat way.

When all the components of the problems, a, r, p (state, action, reward, and transi-tion probability) are known, the algorithm isdenoted “model-based.” Often, one or both ofthe r, p functionals are not known in advance,and have to be learned while solving the prob-lem, usually through repetitive play. This isdenoted as “model-free” learning.

2. Model-free algorithms: If the state transitionprobabilities are not known in advance, thenthe MDP is solved by collecting sample-pathsof the state transitions, which are gener-ated by the “environment” (latent transitionprobabilities), and the corresponding rewards,and then estimating the optimal State–ActionValue Function Q∗(s, a) using statistical aver-aging. Note that the state value function V∗(s)is no longer useful in the model-free case,since even if it were known, the calculationof the optimal policy still requires knowledgeof the state dynamics. On the other hand,once Q∗(s, a) is known, the optimal policycan be obtained by doing a simple maximiza-tion π∗(s) = arg maxaQ∗(s, a). The two mainclasses of Model Free algorithms are:

• Monte Carlo (MC) Learning: These algo-rithms work for cases when the MDPsamplepaths terminate, and proceed by estimat-ing Qπ(s, a) by averaging the total futurerewards seen whenever the system is in states and the agent takes action a. This results inan unbiased estimate of Qπ(s, a), howeverit is subject to a large variance, as a result ofwhich a large number of sample paths areneeded to get a good estimate.

Journal Of Investment Management Second Quarter 2020

Dynamic Goals-Based Wealth Management Using Reinforcement Learning 11

• Temporal Difference (TD) Learning: Thesealgorithms work even for nonterminatingMDPs, and are lower variance and thusmore sample efficient than Monte Carlomethods. They use a one-step version of theBEE given by the following iteration:

Q(s, a)← Q(s, a)+ α[R+ γQ(s′, a′)

−Q(s, a)]. (5)

The reward R, state s, and action a areseen at time t, and the state and action nextperiod are denoted s′ and a′, respectively.The parameter α proxies for the “learningrate” and is usually chosen to be a smallvalue. The “discount rate” is γ ≤ 1 and itsuffices to tradeoff later rewards against ear-lier ones in an episode. It also helps to set ahorizon on the importance of rewards whenepisodes do not terminate in a short horizon.

We seek to obtain estimates of the optimalState-Action Value Function Q∗(s, a) usingthe generated sample paths. When in state s,the action a is generated according to whatis known as an “epsilon-greedy” policy π.Under this ε-greedy approach, the currentaction a is chosen based on the currentpolicy with probability (1−ε)but with prob-ability ε, a random action is chosen. Usingthe current policy is known as “exploita-tion” and using the random policy generates“exploration” behavior. Exploration is akey ingredient in RL, because it enablesbetter coverage of the state space. Thisidea optimally implements the exploitation–exploration tradeoff. Staying on the beatentrack (exploitation) may not lead to the bestsolution and some wandering (exploration)often leads to discovering better outcomes.Note that for our specific portfolio prob-lem, R = 0 except at time T , when it takesa value in {0, 1}. The equation above can

therefore, also be written as

Q(s, a)← Q(s, a)(1− α)

+ α(γ ·Q(s′, a′)) (6)

where we note that this update equation setsthe new value of Q(s, a) to a weighted aver-age of the current value function and thevalue function in the next period, and whenα is small, the learning is of course slow, butconvergence is more stable.

This formula in Equation (5) uses the fol-lowing sample path transitions: Start fromstate s and take action a (using the ε-greedypolicy) to generate reward R, followed bya probabilistic transition to state s′ fromwhere the action a′ is taken, again underthe ε-greedy policy. This is followed by aversion of the policy iteration algorithm,to progressively refine the policy, until itconverges to the optimal policy π∗. TDLearning comes in two flavors:

(a) SARSA-Learning: This is an “on-policy” version of TD Learning, inwhich the policy π being followed togenerate the sample paths is the sameas the current iteration of the optimalpolicy. Note that the current policy maynot be optimal unless it has converged.In Equation (6), both a and a′ are cho-sen using the current policy. Therefore,it is called “on- policy” learning.

(b) Q-Learning: This is an “off-policy” ver-sion of TD Learning, in which thepolicy being used to generate the sam-ple paths (called the “behavior policy”)may not be the same as the current iter-ation of the optimal policy (called the“target policy”). This is a very bene-ficial property to have for two reasons:(1) The behavior policy can be designedto explore more states and actions, thus

Second Quarter 2020 Journal Of Investment Management

12 Sanjiv R. Das and Subir Varma

improving the Q-estimates. Using theoptimal target policy instead to gener-ate sample paths leads to the problemthat not all states and actions will befully explored. (2) Due to the off-policynature of Q-learning, state transitionscan be stored and used multiple timesin order to improve the Q estimates.In contrast on-policy methods need togenerate new sample paths every timethe policy changes. The iteration inQ-Learning is given by:

Q(s, a)← Q(s, a)+ α[R+maxa′

× γQ(s′, a′)−Q(s, a)]In this equation action a is chosenaccording to the behavior policy whilea′ is chosen according to the target pol-icy. See that in Q-Learning the policy a′is chosen optimally from highest valuefunction in state s′, unlike in the caseof SARSA, where it is chosen basedon the current policy function. In thispaper, we implement Q-Learning ona 3D tensor—that is, we implementtabular RL.

3. Algorithms that use function approxima-tors: The reinforcement learning algorithmsdescribed so far are tabular in nature, sincethey work only for discrete values of statesand actions. If this assumption is not satis-fied, or if the number of states (or actions)is extremely large, then these methods do notwork any longer. In their place we have a rangeof methods that use a function approximator,such as a neural network, rather than a table,to represent the value function. This results inthe following two classes of algorithms:

• Deep Q-Learning: In this case a neuralnetwork is used to approximate the state–action value function Qπ(s, a). The neural

network is trained in a supervised fashion,by using training sample paths from theMDP to generate the ground-truth valuesfor Q. Some recent successes of RL, suchas the Atari game playing system devel-oped by DeepMind, were based on the deepQ-Learning algorithm. These are knownas DQNs or deep Q nets, see Mnih et al.(2013).• Policy gradients: This is an alternative

approach to RL, in which the policy isoptimized directly (as opposed to indirectlyobtaining the policy by first estimating valuefunctions). In order to do so, policiesare represented using neural networks, andthe policy optimization proceeds by usingwell-known techniques such as stochasticgradient ascent. Policy gradient methodswork even for cases when the action spaceis continuous and can also accommodaterandomized polices. Applications of RL toareas such as robotics and finance often usepolicy gradients.

Next, we describe the specifics of our algo-rithm for the GBWM problem.

6 Our algorithm

We first solve the problem using classical DPbased on Equation (2). This gives us solutionsagainst which we may check our RL algorithm.We do not provide further details regarding thestandard Bellman approach for DP as it is wellknown.

The algorithm we use solve the MDP is the typeof RL algorithm called tabular Q- Learning and isstated below:

• Define a quality function Q that maps to each ofthe states and actions in the MDP and initializeit to 0—that is, Q(W(t), t, ak) = 0, 0 ≤ t ≤T, 1 ≤ k ≤ K. Note that action ak correspondsto using the efficient frontier pair (μk, σk).

Journal Of Investment Management Second Quarter 2020

Dynamic Goals-Based Wealth Management Using Reinforcement Learning 13

• Set W(0) to a constant corresponding to theinitial wealth at time t = 0.• Initialize time to t = 0 and repeat the follow-

ing steps in a loop for M episodes (or trainingepochs):

1. Choose action a as the one that maximizesQ(W(t), t; ak), 1 ≤ k ≤ K, as modified by theε-greedy algorithm. The ε-greedy approach isone that arbitrarily chooses a random strategywith probability ε to implement the “exploreversus exploit” tradeoff. We describe the exactspecification of the ε-greedy choice in the nextsection.

2. Transition to the next state (W(t + 1), t + 1),where W(t+1) is sampled using the MDP statetransition probability values. While we knowthe exact transition function, we operate as ifthis is generated by the environment and is notknown to the agent.

3. Choose the next action a′ in state (W(t + 1),

t + 1), as the one that maximizes Q(W(t +1), t + 1, ak), 1 ≤ k ≤ K

4. Update the Q value of the original state(W(t), t) and action a, using

Q(W(t), t, a)

← Q(W(t), t, a)+ α[0+ γQ(W(t + 1),

t + 1, a′)−Q(W(t), t, a)].Note that rewards are 0 for intermediate statest < T .

5. t→ t + 1.

6. If t = T , then this is the end of theepisode. Update the Q values for the final state(W(T ), T ):

Q(W(T ), T, a)

← Q(W(T ), T, a)+ α[1−Q(W(T ), T, a)]if W(T ) ≥ H

Q(W(T ), T, a)

← Q(W(T ), T, a)+ α[0−Q(W(T ), T, a)]if W(T ) < H

Increase the number of episodes by 1, set t = 0and re-initialize the state to (W(0), 0) to starta new episode.

7. If t < T then go back to step 1 to continue thecurrent episode.

Both the DP (Planning) algorithm and the Q-Learning algorithm were implemented in thePython programming language. The first com-ponent of the algorithm was to decide the gridfor portfolio wealth outcomes. Below we displayonly some snippets of code in order to make theprogramming of the algorithm clearer.2 Think ofthese snippets as pseudo-code.

1. Create the wealth grid. The code below createsan equally spaced grid in log wealth, which isthen translated back into wealth by exponenti-ation (line 8). W0 is initial wealth. The numberof periods is T (line 4). The infusions at eachtime t are I(t) (lines 5, 6). The grid size wasset to 101 nodes (line 7).

1 lnW = log(W0)

2 lnw_min = lnW

3 lnw_max = lnW

4 for t in range(1,T+1):

5 lnw_min = log(exp(lnw_min)+I[t]) + (mu_min - 0.5* sig*sig)*h

- 3*sig*sqrt(h)

6 lnw_max = log(exp(lnw_max)+I[t]) + (mu_max - 0.5* sig*sig)*h

+ 3*sig*sqrt(h)

7 lnw_array = linspace(lnw_min ,lnw_max ,101)

8 w_array = exp(lnw_array)

Second Quarter 2020 Journal Of Investment Management

14 Sanjiv R. Das and Subir Varma

2. Construct a blank 3D tensor that combines the 2D state space and the 1D action space. This willhold the tabular Q(S, A) function values. The first dimension of the tensor is wealth, the second oneis time (from 0 to T ), and the third is the action space, where NP = K is the number of portfoliosavailable to choose from.

1 Q = zeros((len(w_array), TT+1, NP))

3. Initialize 3D reward tensor in {W, t, a}. We see that rewards are only attained at maturity in thisproblem if the final wealth value is greater than goal level H . We see that the Q and R tensors areof the same dimension.

1 R = zeros((len(w_array), TT+1, NP))

2 for j in range(maxlenW):

3 if W[TT][j]>H:

4 R[j,TT ,:] = 1.0

4. State transition under the policy. Suppose we are at node Wi(t) at time t and transition to a node attime t+ 1, denoted Wj(t+ 1). Which node we transition to depends on the environment (transitionprobabilities), but these in turn depend on the action taken—that is, ak = (μk, σk). We create aseparate function to generate state transitions—that is, to mimic the behavior of the portfolio’swealth from the underlying environment. Given current scalar w0,t0, and action a0, sample atransition to wealth vector w1 at time t1 (line 1). Infusions are denoted by variable I (lines 1, 4).Actiona0 involves the choice of a pair mu, sigma (lines 2, 3). These are drawn from a set of possiblepairs of mean return from list EF_mu and standard deviation of return from list EF_sig. We haveto normalize probabilities from line 4 in line 5. The probabilistic transition under the policy is thenselected in line 6. We return the grid index of wealth in line 7.

1 def sampleWidx(w0 ,w1 ,I,a0): #to give the next state

2 mu = EF_mu[a0]

3 sig = EF_sig[a0]

4 p1 = norm.pdf((log(w1/(w0+I))-(mu -0.5* sig **2)*h)/(sig*sqrt(h

)))

5 p1 = p1/sum(p1) #normalize probabilities

6 idx = where(rand() > p1.cumsum ())[0]

7 return len(idx) #gives index of the wealth node w1 at t1

We may also easily replace the normal distribution with a t-distribution (or any other). For example,line 4 above would be replace with

1 p1 = t.pdf((log(w1/(w0+I))-(mu -0.5* sig **2)*h)/(sig*sqrt(h)) ,5)

where we see that the function norm.pdf is replaced with t.pdf—that is, a t-distribution with5 degrees of freedom.

5. Temporal difference update at a single node. When we arrive at a node in the state space (indexed byidx0, t0 in line 1 below), we then have to pick an action a0, which we do using the epsilon-greedyapproach (lines 2–9). Under that action we will then call the preceding function to ascertain thenext state idx1,t1. We then update the State–Action Value Function Q[idx0,t0,a0] in lines

Journal Of Investment Management Second Quarter 2020

Dynamic Goals-Based Wealth Management Using Reinforcement Learning 15

10–16 if we are at t < T ; or lines 17–19 if we are at t = T . Note that, in line 16, we choose theoptimal policy at t1, as you can see the element Q[idx1,t1,:].max() in the code. In TDLearning, we update at every step in an episode, so it is easy to build all the update logic into asingle generic function for one node, which we call doOneNode(idx0,t0) here.

1 def doOneNode(idx0 ,t0): #idx0: index on the wealth axis , t0:

index on the time axis

2 #Pick optimal action a0 using epsilon greedy approach

3 if rand() < epsilon:

4 a0 = randint(0,NP) #index of action; or plug in best

action from last step

5 else:

6 q = Q[idx0 ,t0 ,:]

7 a0 = where(q==q.max())[0] #Choose optimal Behavior

policy

8 if len(a0) >1:

9 a0 = random.choice(a0) #randint (0,NP) #pick

randomly from multiple maximizing actions

10 #Generate next state S’ at t+1, given S at t and action a0,

and update State -Action Value Function Q(S,A)

11 t1 = t0 + 1

12 if t0 <TT: #at t<T

13 w0 = W[t0][idx0] #scalar

14 w1 = W[t1] #vector

15 idx1 = sampleWidx(w0,w1 ,infusions[t0],a0) #Model -free

transition

16 Q[idx0 ,t0,a0] = Q[idx0 ,t0,a0] + alpha *(R[idx0 ,t0,a0] +

gamma*Q[idx1 ,t1 ,:]. max() - Q[idx0 ,t0,a0])

17 else: #at T

18 Q[idx0 ,t0,a0] = (1-alpha)*Q[idx0 ,t0,a0] + alpha*R[idx0 ,

t0,a0]

19 idx1 = idx0

20 return [idx1 ,t1] #gives back next state (index of W and t)

6. String together a sequence of calls to the previous function to generate updates through one episode,moving forward in time. At the beginning we set idx equal to the wealth index for initial wealth W0.The kernel of the code for one episode is just this. At every point in the episode, whichever state isvisited experiences an update, and the entire Q table evolves into a new policy.

1 for t in range(TT+1):

2 [idx ,t] = doOneNode(idx ,t)

7. We choose the number of episodes (epochs) as 105,000. Other parameters chosen are α = 0.1,γ = 1, and ε = 0.3. We initialize the Q tensor to zeros and then begin processing episode afterepisode. In order to examine if the algorithm is converging to a stable policy, we compute the sumof squared differences between Q tensors from consecutive episodes. At close to 50,000 epochsthis metric becomes very small and stabilizes. Still, we run 55,000 more epochs to be assured ofconvergence.

Second Quarter 2020 Journal Of Investment Management

16 Sanjiv R. Das and Subir Varma

In the next section, we present illustrative resultsfrom a numerical implementation of the Q-Learning algorithm.

7 Experimental results

We present some experimental results from run-ning the Q-Learning algorithm in Table 1. Thetable shows the inputs to the problem, which arethe mean vector of returns and the covariancematrix of returns. These are then used to computethe K = 15 portfolio that is available in the actionspace. The model outputs are the algorithm used,the number of training epochs, and the final valuefunction outcome. We also offer extensions andobservations. We recap some algorithm detailsand note the following:

1. Discussion of the epsilon-greedy algorithm:Q-Learning uses the epsilon-greedy algorithmin order to choose the action (or portfolio inthis case), from the state (W(t), t) at each stepin the episode. This algorithm is as follows:

• Sample x from the Bernoulli DistributionB(ε, 1− e).• If x < ε: Choose action ak, 1 ≤ k ≤ K

with probability 1K

, else: Choose action a =arg max1≤k≤K Q(W(t), t, ak)

Most of the time this algorithm chooses theaction that maximizes the Q value for smallε. However every once in a while it choosesan action uniformly from the set of availableactions. This allows the Q-Learning algorithmto explore states and actions that it otherwisewould not if it were to strictly follow the opti-mal policy. The value of ε has a significanteffect on the working of the algorithm, andhas to be at least 0.30 in order to get goodresults. This shows that without a sufficientamount of exploration, the algorithm may getstuck in states and actions that cause it tounder estimate the Q values. Also note that

the epsilon-greedy policy is being used herein an off-policy fashion, so that larger valuesof epsilon do not affect the accuracy of the Q

values being estimated.2. The results of Q-Learning algorithm are veri-

fied by comparing the value function at t = 0,given by V(W(0), 0), with that computed usingregular DP. Note that we can obtain V readilyfrom Q by using the formula:

V(W(0), 0) = max1≤k≤K

Q(W(0), 0, ak)

By definition V(W(0), 0) is the maximumexpected reward when starting with aninitial wealth of W(0) at t = 0. Inthis case the expected reward correspondsto the probability of the final expectedwealth value exceeding H . Applying regu-lar Dynamic Programming to this problemyields V(W(0), 0) = 0.72, and we can seethat Q-Learning also gives this answer aftertraining for 100 K episodes, provided the valueof the epsilon-greedy parameter is at least0.30. Larger values of epsilon lead to greaterexploration of the state space, which ulti-mately improves the accuracy of the Q values.However this comes at the cost of slower con-vergence, since the algorithm wanders over alarger number of states and actions. This canbe seen in Table 1 for ε = 0.4. In this case thealgorithm converges to a good estimate of theoptimal Q, but takes a larger number of itera-tions to do so. When the RL algorithm is runto a very high number of epochs, say 500 K,then it converges to the DP result, as seen inthe last line of Table 1.

3. Choice of parameters (α, γ): The parameterγ is used in the Q-Learning algorithm as adiscount factor for future rewards. Since thereward used in the GBWM problem formula-tion does not require any discounting, we setγ = 1. The parameter α is used to control thewindow ofQvalues that are averaged together.

Journal Of Investment Management Second Quarter 2020

Dynamic Goals-Based Wealth Management Using Reinforcement Learning 17

Table 1 Results from the Q-Learning Algorithm. The parameters for these runsare as follows. The initial portfolio wealth is W(0) = 100; target portfoliogoal = 200; horizon is T = 10 years. A total of 15 portfolios are used andthese are generated from a mean vector of returns M and a covariance matrixof returns

∑shown below, along with the mean and standard deviation of the

portfolios’ returns derived from M and∑

. The RL algorithm used the followingparameters: α = 0.10; y = 1. We assume zero infusions. The run time for50 K epochs is ∼1.5 minutes and for 100 K epochs is ∼3 minutes. Dynamicprogramming, of course, takes 0.5 seconds. And the solution is provided in thetop row of the bottom panel below.

MODEL INPUTS

M =⎡⎣0.05

0.100.25

⎤⎦;

∑ =⎡⎣0.0025 0 0

0 0.04 0.020 0.02 0.25

⎤⎦

Portfolios

0 1 2 3 4 5 6 7

μ 0.0526 0.0552 0.0577 0.0603 0.0629 0.0655 0.0680 0.0706σ 0.0485 0.0486 0.0493 0.0508 0.0529 0.0556 0.0587 0.0623

Portfolios

8 9 10 11 12 13 14

μ 0.0732 0.0757 0.0783 0.0809 0.0835 0.0860 0.0886σ 0.0662 0.0705 0.0749 0.0796 0.0844 0.0894 0.0945

MODEL OUTPUTS

ε No. of epochs V [W(0), t = 0]DP solution 1 0.720.10 50K 0.650.10 100K 0.650.20 50K 0.690.20 100K 0.710.25 100K 0.710.30 50K 0.720.30 100K 0.720.40 50K 0.730.40 100K 0.770.40 200K 0.750.40 500K 0.71

Second Quarter 2020 Journal Of Investment Management

18 Sanjiv R. Das and Subir Varma

Figure 3 Convergence of the algorithm over successive epochs. The solution is reached in approximately20,000 epochs.

Experimentally we observed that α = 0.1works quite well, which corresponds to amoving average over the last 10 Q values.

4. In order to measure the convergence of thealgorithm, we plot the moving average squareddifference between the Q-tensor from succes-sive epochs. Figure 3 shows that the algorithmstabilizes in about 20,000 epochs.

5. Solving GBWM with other algorithms: Thereare a number of other algorithms that can beused to solve the GBWM problem. Since thestate transition dynamics are specified to fol-low the geometric Brownian motion model,we can apply classical DP algorithms to thisproblem, as shown in Das et al. (2019).RL algorithms are needed for the followingcases in which Dynamic Programming is notapplicable:

• The state transition dynamics are notknown: In this case DP can no longer beused, however Q-Learning is still applica-ble provided there is a collection of samplepaths that can be used for training.

• The state space is not discretized: DP is dif-ficult to implement numerically if we do notdiscretize the state space, and unfortunatelythe tabular Q-Learning algorithm does not

work either. However, continuous states canbe handled by deep Q-Learning using func-tion approximators, or by the policy gradi-ents algorithm. Likewise, continuous-time,continuous-space versions of the Bellman(1952) approach may be used for dynamicprogramming as in Merton (1971).

• The results with the t-distribution are notmuch different than the normal. This sug-gests that the dynamic portfolio solution isrobust to different distributional choices.

8 Concluding comments

DP may be used to solve multiperiod portfo-lio problems to reach desired goals with thehighest possible probability. This is known asGBWM. This paper introduced RL as an alter-nate approach to solving the GBWM problem. Inaddition to providing a brief taxonomy of RLsolu-tion approaches, we also implemented one suchapproach, Q-Learning, and showed that we get thesame results as DP. Our goal is to provide a quickintroduction to how dynamic portfolios may bemodeled using RL. The RL approach is highlyextensible to larger state and action spaces. Forexample, if the action space (portfolios that maybe chosen) varies based on whether the economy

Journal Of Investment Management Second Quarter 2020

Dynamic Goals-Based Wealth Management Using Reinforcement Learning 19

is in normal times or in a recession, then it adds thestate of the economy as an additional dimensionto the problem. This can be easily handled withRL. RL especially shines in comparison to DPwhen the problem becomes path-dependent, suchas in the case of multiperiod portfolio optimiza-tion with taxes, when keeping track of the costbasis across portfolio holdings is required and thistax basis depends on the path of the portfolio, sothat classic DP via backward recursion becomescomputationally expensive from an explosion inthe state space.

The recent advances in hardware and software forRLsuggest great potential for finance applicationsthat depend on dynamic optimization in stochasticenvironments whose transition probabilities arehard to estimate, such as high-frequency trad-ing (HFT). HFT has been one of the areas ofearly investigation of RL in finance. There is along history of papers implementing RL modelsfor trading, such as Moody and Saffell (2001),Dempster and Leemans (2006), Li et al. (2007),Lu (2017), Du et al. (2018), and Zarkias et al.(2019). Additional areas in which RL may beused are option pricing (Halperin, 2017), optimalhedging of derivatives (Halperin, 2018), market-making agents (Halperin and Feldshteyn (2018),Zarkias et al. (2019), cryptocurrencies, optimaltrade execution, etc.

Notes1 The latest (2018) version of this book is available here:

http://incompleteideas.net/book/the-book-2nd.html.2 If you wish to implement the code, you will need to wrap

these code ideas into a full Python program.

References

Bellman, R. (1952). “On the Theory of Dynamic Program-ming,” Proceedings of the National Academy of Sciences38(8), 716–719.

Bellman, R. E. (2003). Dynamic Programming (New York,NY, USA: Dover Publications, Inc.)

Bellman, R. E. and Dreyfus, S. E. (2015). Applied DynamicProgramming (Princeton University Press).

Brunel, J. (2015). Goals-Based Wealth Management: AnIntegrated and Practical Approach to Changing the Struc-ture of Wealth Advisory Practices (New York: Wiley).

Chhabra, A. B. (2005). “Beyond Markowitz: A Com-prehensive Wealth Allocation Framework for IndividualInvestors,” The Journal of Wealth Management 7(4),8–34.

Das, S. R., Markowitz, H., Scheid, H. J., and Statman, M.(2010). “Portfolio Optimization with Mental Accounts,”Journal of Financial and Quantitative Analysis 45(2),311–334.

Das, S. R., Ostrov, D., Radhakrishnan, A., and Srivastav,D. (2018). “Goals-Based Wealth Management: A NewApproach,” Journal of Investment Management 16(3),1–27.

Das, S. R., Ostrov, D., Radhakrishnan, A. and Srivas-tav, D. (2019). “A Dynamic Approach to Goals-BasedWealth Management,” Computational Management Sci-ence, https://doi.org/10.1007/s10287-019-06351-7.

Dempster, M. A. H. and Leemans, V. (2006). “An Auto-mated FX Trading System Using Adaptive Reinforce-ment Learning,” Expert Systems with Applications 30(3),543–552.

Du, X., Zhai, J. and Koupin, L. (2018). “AlgorithmTrading Using Q-Learning and Recurrent ReinforcementLearning,” Working Paper, Stanford University.

Halperin, I. (2017). QLBS: Q-Learner in the Black-Scholes(-Merton) Worlds, SSRN Scholarly Paper ID 3087076,Social Science Research Network, Rochester, NY.

Halperin, I. (2018). The QLBS Q-Learner Goes NuQLear:Fitted Q Iteration, Inverse RL, and Option Portfolios,SSRN Scholarly Paper ID 3102707, Social ScienceResearch Network, Rochester, NY.

Halperin, I. and Feldshteyn, I. (2018). Market Self-learningof Signals, Impact and Optimal Trading: Invisible HandInference with Free Energy (Or, How We Learned toStop Worrying and Love Bounded Rationality). SSRNScholarly Paper ID 3174498, Social Science ResearchNetwork, Rochester, NY.

Li, H., Dagli, C. H., and Enke, D. (2007). “Short-term StockMarket Timing Prediction under Reinforcement LearningSchemes,” In 2007 IEEE International Symposium onApproximate Dynamic Programming and ReinforcementLearning, pp. 233–240.

Lu, D. W. (2017). “Agent Inspired Trading Using RecurrentReinforcement Learning and LSTM Neural Networks,”arXiv:1707.07338 [q-fin].

Second Quarter 2020 Journal Of Investment Management

20 Sanjiv R. Das and Subir Varma

Markowitz, H. H. (1952). “Portfolio Selection,” Journal ofFinance 6, 77–91.

Merton, R. (1969). “Lifetime Portfolio Selection underUncertainty: The Continuous-Time Case,” The Reviewof Economics and Statistics 51(3), 247–257.

Merton, R. C. (1971). “Optimum Consumption and Port-folio Rules in a Continuous-Time Model,” Journal ofEconomic Theory 3(4), 373–413.

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A.,Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013).“Playing Atari with Deep Reinforcement Learning,”arXiv:1312.5602 [cs].

Moody, J. and Saffell, M. (2001). “Learning to Trade viaDirect Reinforcement,” IEEE Transactions on NeuralNetworks 12(4), 875–889.

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou,I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai,M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre,

L., van den Driessche, G., Graepel, T., and Hassabis,D. (2017). “Mastering the Game of Go without HumanKnowledge,” Nature 550(7676), 354–359.

Sutton, R. S. and Barto, A. G. (1998). ReinforcementLearning: An Introduction (second edition edition ed.).Cambridge, Mass: A Bradford Book.

Zarkias, K. S., Passalis, N., Tsantekidis, A. and Tefas,A. (2019). “Deep Reinforcement Learning for FinancialTrading Using Price Trailing,” In ICASSP 2019—2019IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP), pp. 3067–3071.

Keywords:

Journal Of Investment Management Second Quarter 2020