Dynamic Programming

2 31 2 3

1

We are often interested in net present values:

...

Net present values can be expressed as the sum of a current payoff and a discounted

future value:

Often we

t t t t tt

t t t

V

V V

can eliminate time subscripts by establishing that value each period depends

on a set of "state variables" that describe different "states of the world":

( ) ( , ') ( ')

where ' depends on

This yi

V k k k V k

k k

elds a "stationary" problem (one that is not time-dependent)

Dynamic Programming is a set of tools and techniques for analyzing such problems

A Simple Example

0{ , } 0

1

Consider a simple economy with a representative agent who lives forever

The agent's problem is:

max ( )

subject to

t t

tt

c i t

t

U c

k

(1 )

( )

where is the discount factor, (.) is the utility function, is consumption

in period , is capital in period , is investment

t t

t t t

t

t t

k i

c i F k

U c

t k t i

1 1

in period , and (.) is the

production function

We can use the constraints to eliminate and express as

( ) (1 ) ( )

t t

t t t t t t

t F

i c

c F k k k f k k

Capital as a State Variable

1 01

{ } 0

1

The agent's problem can be rewritten as

max ( ( ) )

subject to 0 ( ), 0,1,2,....

t t

tt t

k t

t t

U f k k

k f k t

0 0 is given

Since this is an infinite horizon problem, it looks the same every period; the only thing

that differs if you begin in another period

k

0

is the initial capital stock

Thus, intuitively, the solution to this problem must depend entirely on

Another feature of this problem: if we start in period 0, the agent does not need to

choose the en

k

tire sequence of capital stocks immediately

The agent just needs to choose the next period's capital stock; this is sufficient to

determine current consumption and next period's output

The agent can wait 2

1

until period 1 to choose , and so on

Given this, we just need to describe the agent's preferences over given t t

k

k k

The Value Function

0

0

0

Suppose that the agent's problem has been solved for every value of

Then we can define a ( ) that represents the value of the maximized

objective function for each :

k

value function V k

k

1 0

0 10

0

00 (

( ) ( ( ) )

when the sequence of capital stocks is chosen optimally subject to the constraints

Now for any we can say

( ) max

tt t

t

k f k

V k U f k k

k

V k

0 1 1)

( ( ) ) ( )

The time subscripts are now completely unnecessary: the problem is , or

independent of the date

U f k k V k

stationary

The Functional Equation

0 ' ( )

In general, given a current capital stock ,

( ) max ( ( ) ') ( ')

This one equation in the unknown function (.) is called a

The study of

k f k

k

V k U f k k V k

V functional equation

1

dynamic optimization problems through the analysis of such functional

equations is called

We can often describe the optimal choices using a (.)

The relationship t

dynamic programming

policy function g

k ( ) completely describes the dynamics of capital

accumulation (changes in the state variable) from any given initial statetg k

Computing the Value Function

0 ' ( )

Note that in the problem

( ) max ( ( ) ') ( ')

(.) and (.) are known

The unknown is the value function

In general there are three ways to solve for the value

k f kV k U f k k V k

U f

function:

1. For a few simple cases, ( ) has a known functional form (if you pick the right

(.) and (.), a textbook can tell you what ( ) is)

2. You can guess. This method is notoriously unreliable

3

V k

U f V k

. You can compute ( ) numerically on a computer. This requires that you choose

particular functional forms and parameter values, and then use an appropriate

computational technique

V k

Computing the Value Function

The standard approach for computing value functions is known as “iteration on the value function”

1. To proceed, first specify functional forms and parameter values for the primitives of the problem: utility functions, production functions, the discount factor, etc.

2. Then set up a grid of possible values of the state variable: in cases where the state variable is continuous (such as the capital accumulation problem) this results in an approximate solution rather than an exact one, but this is usually good enough

3. The next step is to choose an initial value function; this sounds difficult but in practice it usually doesn’t matter too much what you start with: setting V(k) = 0 for every k on the grid is a typical choice

Iteration on the Value Function

0 k' f(k)

4. At every grid point , taking the initial function (.) as given, choose the value

of ' that solves the agent's problem:

max ( ( ) ') ( ')

Reco

k V

k

U f k k V k

rd the value of ( ( ) ') ( ') as ( ) in a new vector of values

5. Repeat step 4. using the new (.) function; keep repeating until (.) no longer

changes (at which point we say that the value funct

U f k k V k V k

V V

ion has converged).

Mathematical Foundations

• We would like our model to yield a unique solution for V(.) (a unique function, not a point)

• This means no matter which initial value function we choose at the start of the iterations, the final value function is always the same

• Similarly, we would like g(.) to be single-valued (a function instead of a correspondence)

• Then, given an initial state, the model would yield a unique prediction for the entire path of capital accumulation

• Whether the model yields unique solutions for the value function and policy function depends on the primitives of the problem: utility functions, production functions, etc.

• Stokey and Lucas describe general sufficient conditions for a model to yield a unique value function and a unique policy function (these conditions are sufficient, not necessary)

General Notation

( )

We'll follow the notation in Stokey and Lucas (1989) Section 4.2 and consider

functional equations of the form:

( ) max[ ( , ) ( )]

under the assumption that i

y xv x F x y v y

F

s bounded and 1

is the set of possible values of the state variable

: is the correspondence describing the feasibility constraints

{( , ) : ( )} is the graph of

: is the return functi

X

X X

A x y X X y x

F A R

on

[0,1) is the discount factor

Assumptions

4.3: is a convex subset of , and the correspondence : is non-empty,

compact-valued, and continuous.

4.4: The function : is bounded and continuous, and (0,1).

4.7: is strictly concave.

4.8:

lX R X X

F A R

F

is convex in the sense that for any [0,1], and , ' , if ( ) and

' ( '), then (1 ) ' ( (1 ) ').

x x X y x

y x y y x x

Key Results

Given assumptions 4.3, 4.4, 4.7, and 4.8, there is a unique bounded and continuous

function that satisfies the functional equation. Further, is strictly concave, and the

associated optimal policy fu

v v

0 1

nction is single-valued and continuous.

Given , the sequence { } defined by ( ), 0,1,2,... is the unique

solution to the dynamic programming problem.

We'll go over the assumptions to establ

t t t

g

x x x g x t

ish what might go wrong if they are not

satisfied, but it is important to keep in mind that the assumptions are sufficient

conditions, not necessary ones. It is possible to use dynamic programming in

other environments. "Existence proof by computer" is fairly common.

The Content of the Assumptions

To understand most of the assumptions, we do not really need to think about

dynamic programming.

It is sufficient to think about any constrained optimization problem.

In a constrained optimization problem, the constraints must allow for at least

one solution. Thus, ( ) must be non-empty.

The constraints must also create a feasible set that is closed and bounded.

Closed is necessary because you cannot m

x

aximize on an open interval.

To see this, consider max subject to [0,1).

Bounded is necessary so that the objective function does not reach

Closed and bounded compact, so ( ) must be compact-val

x x

x

ued.

The Assumptions

Assuming that is bounded helps ensure that is bounded

A bounded feasible set is insufficient for this: consider max 1/ s.t. [0,1]

The solution is 0

Assuming that F is continuous is important for

F v

x x

x

x

obtaining a max

If the objective function is not continuous, a max may not be obtainable

Consider ( ) 0 if 0 and 1/ if 0

If the problem is max ( ) s.t. [0,1] then there is no solution

f x x x x

f x x

The Assumptions

Note that in the value maximization problem the objective function is

( , ) ( )

In order to ensure that this function is continuous, we need ( ) to be continuous

Otherwise, slight changes in could

F x y v y

x

x

t

0

lead to jumps in that could cause

( ) to have breaks

We need 1 because otherwise the infinite sum ( , )

may not be bounded even if ( , ) ist

y

v x

F x y

F x y

The Assumptions

In order to obtain a unique solution for every , we need to ensure that

the objective function is strictly concave

This rules out flat segments or multiple peaks

In order to attain a single-peak and

y x

2

x

have a unique solution, the constraint

set must be convex and must be convex

To see why, consider the problem max{ } s.t. 1 or 1

X

x x x