On the numerical solution of high-dimensional optimal control...

Transcript of On the numerical solution of high-dimensional optimal control...

On the numerical solution of high-dimensionaloptimal control problems: approximate dynamic

programming and Smolyak's algorithm

Markus Fischer

Humboldt University Berlin / University of Heidelberg

Torino, MSF 2008
Approximate DP and Smolyak's algorithm

Introduction

Introduction

Classical control problems and approximate DP

Smolyak's algorithm

A posteriori error bounds

Bibliography

Introduction

Aim and scope

Procedure for numerical solution of high-dimensionalcontinuous-time optimal control problems.

Combination of methods:

I Approximate dynamic programming: dynamic programming indiscrete time with continuous state space using a method forfunction approximation.

I Smolyak's algorithm: method for constructing interpolation (orintegration) operators for multivariate functions.

I Computation of a posteriori error bounds using arepresentation of value functions due to [Rogers, 2007].

Introduction

Aim and scope






Introduction

Aim and scope






Introduction

Aim and scope






Introduction

Aim and scope






Apart from classical problems, motivation from the numericalsolution of optimal control problems with delay.

Introduction

Numerical solution of continuous-time optimal controlproblems

Standard approach:

I Replace original problem by a sequence of approximatingcontrol problems,

I construct approximating control problems by discretising timeand state space of the original dynamics and costs,

I solve discrete problems by a backward iteration of dynamicprogramming type.

Introduction


Standard approach:




Introduction


Standard approach:




Introduction


Standard approach:




Last step computationally di�cult when the state space ishigh-dimensional, where �high� means dimensions greater thanthree or four � curse of dimensionality.


Introduction


Smolyak's algorithm


Bibliography


Classical stochastic optimal control problemDynamics given by a controlled SDE:

dX (t) = b (t,X (t), u(t)) dt + σ (t,X (t), u(t)) dW (t), t ≥ t0,

with initial condition X (t0) = x ∈ Rd , where W (.) is a d1-dimensionalstandard Wiener process, u(.) ∈ U a control process with values in aspace of control actions Γ, and b : [0,∞)× Rd × Γ → Rd ,σ : [0,∞)× Rd × Γ → Rd×d1 are the drift and di�usion coe�cient.

Cost functional over a �nite time horizon T :

J(t0, x ; u(.)) := E

[∫ Tt0

f (t,X (t), u(t)) dt + g (X (T ))

].

Corresponding value function:

V (t0, x) := infu(.)∈U

J(t0, x ; u(.)), t0 ∈ [0,T ], x ∈ Rd .






J(t0, x ; u(.)) := E

[∫ Tt0

f (t,X (t), u(t)) dt + g (X (T ))

].


V (t0, x) := infu(.)∈U

J(t0, x ; u(.)), t0 ∈ [0,T ], x ∈ Rd .






J(t0, x ; u(.)) := E

[∫ Tt0

f (t,X (t), u(t)) dt + g (X (T ))

].


V (t0, x) := infu(.)∈U

J(t0, x ; u(.)), t0 ∈ [0,T ], x ∈ Rd .


Classical control problem and time discretisation

Regularity assumptions on b, σ, f , g : 12-Hölder in time, Lipschitz in the

space variable, continuous and bounded.

Then V is bounded, 12-Hölder

continuous in time, Lipschitz continuous in space, but not necessarilydi�erentiable.

Euler discretisation of dynamics and costs: time step h := T/N,piecewise constant Γ-valued control processes ū ∈ Ū . Costs given by

J̄(n h, x , ū) := E

N−1∑j=n

fj(X̄ (j), ū(j)

)+ g

(X̄ (N)

) ∣∣∣X̄ (n) = x ,

where fj(x , γ) := h · f (j h, x , γ), X̄ controlled Markov chain withtransition probabilities µj(x , γ) d -variate normal distributions with meanx + h · b(j h, x , γ) and covariance h · (σσT)(j h, x , γ).

Error bounds: [Krylov, 1999] and more recent works.




space variable, continuous and bounded. Then V is bounded, 12-Hölder



J̄(n h, x , ū) := E

N−1∑j=n

fj(X̄ (j), ū(j)

)+ g

(X̄ (N)

) ∣∣∣X̄ (n) = x ,








Euler discretisation of dynamics and costs: time step h := T/N,piecewise constant Γ-valued control processes ū ∈ Ū .

Costs given by

J̄(n h, x , ū) := E

N−1∑j=n

fj(X̄ (j), ū(j)

)+ g

(X̄ (N)

) ∣∣∣X̄ (n) = x ,









J̄(n h, x , ū) := E

N−1∑j=n

fj(X̄ (j), ū(j)

)+ g

(X̄ (N)

) ∣∣∣X̄ (n) = x ,









J̄(n h, x , ū) := E

N−1∑j=n

fj(X̄ (j), ū(j)

)+ g

(X̄ (N)

) ∣∣∣X̄ (n) = x ,




Approximate dynamic programming

One-step Bellman operators Tn : B(Rd) → B(Rd), n ∈ {0, . . . ,N−1}:

Tn(v)(x) := infγ∈Γ

{fn(x , γ) +

∫Rd

v(y) µn(x , γ)(dy)

}

Recall that Tn is non-expansive under the supremum norm, that is,‖Tn(v)− Tn(w)‖∞ ≤ ‖v − w‖∞ for all v ,w ∈ B(Rd). Moreover, Tnpreserves Lipschitz continuity.

According to Bellman's principle: V̄ := V̄ (0, .) = T0 ◦ . . . ◦ TN−1(g).

Approximate DP: Let A : B(Rd) → B(Rd) be linear. SetṼ := A ◦ T0 ◦ . . . ◦ A ◦ TN−1(g).

Then‖V̄ − Ṽ ‖∞ ≤ N · sup

w∈F‖A(w)− w‖∞,

where F contains {Tj−1 ◦ A ◦ Tj ◦ . . . ◦ A ◦ TN−1(g) | j ∈ {1, . . . ,N}}.





{fn(x , γ) +

∫Rd


}Recall that Tn is non-expansive under the supremum norm, that is,‖Tn(v)− Tn(w)‖∞ ≤ ‖v − w‖∞ for all v ,w ∈ B(Rd).

Moreover, Tnpreserves Lipschitz continuity.




w∈F‖A(w)− w‖∞,






{fn(x , γ) +

∫Rd


}Recall that Tn is non-expansive under the supremum norm, that is,‖Tn(v)− Tn(w)‖∞ ≤ ‖v − w‖∞ for all v ,w ∈ B(Rd). Moreover, Tnpreserves Lipschitz continuity.




w∈F‖A(w)− w‖∞,






{fn(x , γ) +

∫Rd






w∈F‖A(w)− w‖∞,






{fn(x , γ) +

∫Rd






w∈F‖A(w)− w‖∞,






{fn(x , γ) +

∫Rd






w∈F‖A(w)− w‖∞,



Approximate dynamic programming II

Approximation error measured in supremum norm; A in this waycompatible with Bellman operators. More complicated for operatorswhich yield good approximations only in Lp-norm [Munos, 2007].

Truncation of state space: from now on, work with [0, 1]d (or[a, b]d ) instead of Rd .

Instead of one approximation operator A, family of operatorsA(q, d), q ≥ d , such that

supw∈F

‖A(q, d)(w)− w‖∞q→∞→ 0

fast enough for suitable function classes F . Use Smolyak'salgorithm for constructing the A(q, d).






supw∈F

‖A(q, d)(w)− w‖∞q→∞→ 0







supw∈F

‖A(q, d)(w)− w‖∞q→∞→ 0



The curse of dimensionality for function approximation

Lipschitz continuous functions:Let Fd := {f : [0, 1]d → R | |f (x)− f (y)| ≤ max |xi − yi |}. Then

errApp(Fd , n) := infΨ=Ψ(n)

supf ∈Fd

‖Ψ(f )− f ‖∞ ≈1

2n−1/d .



Lipschitz continuous functions: � Curse!Let Fd := {f : [0, 1]d → R | |f (x)− f (y)| ≤ max |xi − yi |}. Then


supf ∈Fd

‖Ψ(f )− f ‖∞ ≈1

2n−1/d .





supf ∈Fd

‖Ψ(f )− f ‖∞ ≈1

2n−1/d .

Hölder classes of functions:Let k ∈ N0, β ∈ (0, 1]. Set

Ck,βd := {f : [0, 1]

d → R | |Dαf (x)− Dαf (y)| ≤ max |xi − yi |β , |α| = k}.

Then there are constants 0 < cd < Cd < ∞ such that

cd · n−(k+α)/d ≤ errApp(Ck,βd , n) ≤ Cd · n−(k+α)/d .





supf ∈Fd

‖Ψ(f )− f ‖∞ ≈1

2n−1/d .

Hölder classes of functions: � curse unless k ∼ dLet k ∈ N0, β ∈ (0, 1]. Set

Ck,βd := {f : [0, 1]

d → R | |Dαf (x)− Dαf (y)| ≤ max |xi − yi |β , |α| = k}.

Then there are constants 0 < cd < Cd < ∞ such that

cd · n−(k+α)/d ≤ errApp(Ck,βd , n) ≤ Cd · n−(k+α)/d .

Smolyak's algorithm

Introduction


Smolyak's algorithm


Bibliography

Smolyak's algorithm

Smolyak's algorithm

Method for constructing operators for the approximation (orintegration) of d -variate functions using certain tensor products ofunivariate operators [Smolyak, 1963]:

Let U i , i ∈ N, be a sequence of continuous linear operators. Set∆i := U i − U i−1, U0 := 0.

Smolyak's algorithm in dimension d of degree q−d is de�ned asthe operator

A(q, d) :=∑

~i∈Nd ,|~i |≤q

(∆i1 ⊗ . . .⊗∆id

), q ≥ d ,

where |~i | := i1 + . . . + id for ~i ∈ Nd .

Smolyak's algorithm

Smolyak's algorithm

Method for constructing operators for the approximation (orintegration) of d -variate functions using certain tensor products ofunivariate operators [Smolyak, 1963]:

Let U i , i ∈ N, be a sequence of continuous linear operators. Set∆i := U i − U i−1, U0 := 0.

Smolyak's algorithm in dimension d of degree q−d is de�ned asthe operator

A(q, d) :=∑

~i∈Nd ,|~i |≤q

(∆i1 ⊗ . . .⊗∆id

), q ≥ d ,

where |~i | := i1 + . . . + id for ~i ∈ Nd .

Smolyak's algorithm

Smolyak's algorithm II

Dimension one: A(q, 1) =∑q

j=1 ∆j = Uq, q ≥ 1.

Dimension two:

A(q, 2) =q−1∑j1=1

∑j2=1

∆j1 ⊗∆j2 , q ≥ 2,

while

Uq−1 ⊗ Uq−1 =q−1∑j1=1

q−1∑j2=1

∆j1 ⊗∆j2 .

Hierarchical structure:

A(q, d) = A(q−1, d) +∑

~i∈Nd ,|~i |=q

(∆i1 ⊗ . . .⊗∆id

), q ≥ d .

Smolyak's algorithm



j=1 ∆j = Uq, q ≥ 1.

Dimension two:

A(q, 2) =q−1∑j1=1

q−j1∑j2=1

∆j1 ⊗∆j2 , q ≥ 2,

while

Uq−1 ⊗ Uq−1 =q−1∑j1=1

q−1∑j2=1

∆j1 ⊗∆j2 .


A(q, d) = A(q−1, d) +∑

~i∈Nd ,|~i |=q

(∆i1 ⊗ . . .⊗∆id

), q ≥ d .

Smolyak's algorithm



j=1 ∆j = Uq, q ≥ 1.

Dimension two:

A(q, 2) =q−1∑j1=1

q−j1∑j2=1

∆j1 ⊗∆j2 , q ≥ 2,

while

Uq−1 ⊗ Uq−1 =q−1∑j1=1

q−1∑j2=1

∆j1 ⊗∆j2 .


A(q, d) = A(q−1, d) +∑

~i∈Nd ,|~i |=q

(∆i1 ⊗ . . .⊗∆id

), q ≥ d .

Smolyak's algorithm



j=1 ∆j = Uq, q ≥ 1.

Dimension two:

A(q, 2) =q−1∑j1=1

q−j1∑j2=1

∆j1 ⊗∆j2 , q ≥ 2,

while

Uq−1 ⊗ Uq−1 =q−1∑j1=1

q−1∑j2=1

∆j1 ⊗∆j2 .


A(q, d) = A(q−1, d) +∑

~i∈Nd ,|~i |=q

(∆i1 ⊗ . . .⊗∆id

), q ≥ d .

Smolyak's algorithm

Grid-based interpolation

Let U i , i ∈ N, be operators on univariate functions of the form

U i (f ) =mi∑j=1

f (x ij ) aij ,

x ij ∈ X i ⊂ [0, 1] grid points, aij basis functions, j ∈ {1, . . . ,mi}.

Then A(q, d) is determined by function values on the sparse grid

H(q, d) :=⋃

~i∈Nd ,|~i |≤q

X i1 × . . .× X id , q ≥ d .

Assumption: nested grids, i. e. X i−1 ⊂ X i for all i ∈ N. Then

H(q, d) =⋃|~i |=q

X i1 × . . .× X id =⋃|~i |≤q

∆X i1 × . . .×∆X id ,

where ∆X i := X i \ X i−1, X 0 := ∅.

Smolyak's algorithm



U i (f ) =mi∑j=1

f (x ij ) aij ,

x ij ∈ X i ⊂ [0, 1] grid points, aij basis functions, j ∈ {1, . . . ,mi}.Then A(q, d) is determined by function values on the sparse grid

H(q, d) :=⋃

~i∈Nd ,|~i |≤q

X i1 × . . .× X id , q ≥ d .


H(q, d) =⋃|~i |=q

X i1 × . . .× X id =⋃|~i |≤q

∆X i1 × . . .×∆X id ,

where ∆X i := X i \ X i−1, X 0 := ∅.

Smolyak's algorithm



U i (f ) =mi∑j=1

f (x ij ) aij ,


H(q, d) :=⋃

~i∈Nd ,|~i |≤q

X i1 × . . .× X id , q ≥ d .


H(q, d) =⋃|~i |=q

X i1 × . . .× X id =⋃|~i |≤q

∆X i1 × . . .×∆X id ,

where ∆X i := X i \ X i−1, X 0 := ∅.

Smolyak's algorithm



U i (f ) =mi∑j=1

f (x ij ) aij ,


H(q, d) :=⋃

~i∈Nd ,|~i |≤q

X i1 × . . .× X id , q ≥ d .


H(q, d) =⋃|~i |=q

X i1 × . . .× X id =⋃|~i |≤q

∆X i1 × . . .×∆X id ,

where ∆X i := X i \ X i−1, X 0 := ∅.

Smolyak's algorithm

Piecewise multilinear interpolation on sparse grids

Uniform one-dimensional grids: x11 := 1/2,

x ij := (j−1)/(mi−1) for i > 1, j ∈ {1, . . . ,mi},

where m1 := 1, mi := 2i−1 + 1 if i > 1 (Clenshaw-Curtis).

Piecewise linear univariate basis functions: a11 ≡ 1,

aij(t) :=(1− (mi−1) · |t−x ij |

)· 1[x i

j−1/(mi−1),x ij +1/(mi−1)]

(t)

t ∈ [0, 1], i > 1, j ∈ {1, . . . ,mi}.

Denote by A(q, d) the Smolyak algorithm in dimension d of degreeq−d based on piecewise linear univariate interpolation operators.Denote by H(q, d) the corresponding sparse grid.

Smolyak's algorithm



x ij := (j−1)/(mi−1) for i > 1, j ∈ {1, . . . ,mi},



aij(t) :=(1− (mi−1) · |t−x ij |

)· 1[x i

j−1/(mi−1),x ij +1/(mi−1)]

(t)

t ∈ [0, 1], i > 1, j ∈ {1, . . . ,mi}.


Smolyak's algorithm



x ij := (j−1)/(mi−1) for i > 1, j ∈ {1, . . . ,mi},



aij(t) :=(1− (mi−1) · |t−x ij |

)· 1[x i

j−1/(mi−1),x ij +1/(mi−1)]

(t)

t ∈ [0, 1], i > 1, j ∈ {1, . . . ,mi}.


Smolyak's algorithm

Sparse grids from one-dimensional uniform grids

H(2 + 5, 2) sparse grid of degree 5, levels from 0 to 5:

Smolyak's algorithm

Number of points of the sparse grids

Denote by n(q, d) the number of points of the sparse grid H(q, d),q ≥ d . Then (e. g. [Schreiber, 2000])

max

{(q

d

), 2q−2d

}≤ n(q, d) ≤ 2

−(d−1)

(d − 1)!· 2q · qd−1.

In particular, q ≤ 2d + log2(n(q, d)).

Smolyak's algorithm

Number of points of the sparse grids

Denote by n(q, d) the number of points of the sparse grid H(q, d),q ≥ d . Then (e. g. [Schreiber, 2000])

max

{(q

d

), 2q−2d

}≤ n(q, d) ≤ 2

−(d−1)

(d − 1)!· 2q · qd−1.

In particular, q ≤ 2d + log2(n(q, d)).

Smolyak's algorithm

Hierarchical construction of the interpolant

Let q > d .

A(q, d)(f ) = A(q−1, d)(f ) +∑|~i |=q

(∆i1 ⊗ . . .⊗∆id

)(f )

= A(q−1, d)(f ) +∑|~i |=q

∑~j

β~i~j· ai1j1 ⊗ . . .⊗ a

idjd

,

where ~j ranges over ~j = (j1, . . . , jd ), 1 ≤ jl ≤ #∆X il , and

β~i~j

:= f (x i1j1 , . . . , xidjd

)− A(q−1, d)(f )(x i1j1 , . . . , xidjd

).

Of course,

ai1j1 ⊗ . . .⊗ aidjd

(x) = ai1j1(x1) · . . . · aidjd

(xd ), x = (x1, . . . , xd ) ∈ Rd .

Smolyak's algorithm

Approximation error

The space of functions with bounded mixed partial derivatives up toorder k on [0, 1]d is given by

F kd := {f : [0, 1]d → R | ‖f ‖Fkd

< ∞},

where ‖f ‖Fkd

:= maxα∈Nd0,|α|∞≤k‖D

αf ‖∞.

Important properties: ‖.‖Fkd

is a tensor norm, ‖.‖Fk1

induces a tensor

norm on F k1⊗ . . .⊗ F k

1such that cl(F k

1⊗ . . .⊗ F k

1) ' F kd .

Let k ∈ {1, 2}. Then there is a constant c(d) such that

‖A(q, d)(f )− f ‖∞ ≤ c(d) · ‖f ‖Fkd

· n−k · (log2(n))(k+1)(d−1)

for all q > d , where n = n(q, d) is the number of grid points;

[Smolyak, 1963], [Barthelmann et al., 2000], [Schreiber, 2000].

Smolyak's algorithm

Approximation error


F kd := {f : [0, 1]d → R | ‖f ‖Fkd

< ∞},

where ‖f ‖Fkd


αf ‖∞.



induces a tensor


1such that cl(F k

1⊗ . . .⊗ F k

1) ' F kd .


‖A(q, d)(f )− f ‖∞ ≤ c(d) · ‖f ‖Fkd

· n−k · (log2(n))(k+1)(d−1)



Smolyak's algorithm

Approximation error


F kd := {f : [0, 1]d → R | ‖f ‖Fkd

< ∞},

where ‖f ‖Fkd


αf ‖∞.



induces a tensor


1such that cl(F k

1⊗ . . .⊗ F k

1) ' F kd .


‖A(q, d)(f )− f ‖∞ ≤ c(d) · ‖f ‖Fkd

· n−k · (log2(n))(k+1)(d−1)




Introduction


Smolyak's algorithm


Bibliography


Approximate DP and Smolyak's algorithmLet

Fg :=⋃q≥d

{Tj−1 ◦ A(q, d) ◦ Tj ◦ . . . ◦ A(q, d) ◦ TN−1(g) | j = 1, . . . ,N} .

Compute Ṽ = Ṽ (q) by approximate DP using A(q, d). Then

‖V̄ − Ṽ ‖∞ ≤ N · supw∈Fg

‖A(q, d)(w)− w‖∞.

If we had Fg ⊂ F 1d , then, with n = n(q, d), it would follow that

‖V̄ − Ṽ ‖∞ ≤ N · c(d) · supw∈Fg

‖w‖F 1d· n−1 · (log2(n))2(d−1).

However, functions in Fg are guaranteed only to be Lipschitz.Therefore, a priori worst-case error bound of order n−1/d times alogarithmic factor � curse of dimensionality again!



Fg :=⋃q≥d



‖V̄ − Ṽ ‖∞ ≤ N · supw∈Fg

‖A(q, d)(w)− w‖∞.


‖V̄ − Ṽ ‖∞ ≤ N · c(d) · supw∈Fg

‖w‖F 1d· n−1 · (log2(n))2(d−1).




Fg :=⋃q≥d



‖V̄ − Ṽ ‖∞ ≤ N · supw∈Fg

‖A(q, d)(w)− w‖∞.


‖V̄ − Ṽ ‖∞ ≤ N · c(d) · supw∈Fg

‖w‖F 1d· n−1 · (log2(n))2(d−1).



Upper and lower bound for �xed initial condition

Apply approximate DP with Smolyak's algorithm A(q, d) tocompute Ṽ (j , .), j ∈ {0, . . . ,N}, in particular Ṽ = Ṽ (0, .). Then Ṽis a candidate for V̄ , the value function at time zero of thesemi-discrete problem.

Let x̂ ∈ Rd be any initial state of interest. Use Ṽ to compute anupper as well as a lower bound on V̄ (x̂), the unknown true value.

Upper and lower values by Monte Carlo simulation.


Upper bound: feedback control

Assumption: Γ is compact.

Compute, by Monte Carlo forward simulation of the underlyingMarkov chain with X̄ (0) = x̂ ,

J̄(0, x̂ , ũ) = E

N−1∑j=n

fj(X̄ (j), ũ(j , X̄ (j))

)+ g

(X̄ (N)

) ,where ũ is the feedback control obtained from Ṽ as

ũ(j , x) := argminγ∈Γ

{fj(x , γ) +

∫Rd

Ṽ (y) µj(x , γ)(dy)

}.

Evaluation of ũ only at points of the simulated trajectories.

Then, by de�nition, V̄ (x̂) ≤ J̄(0, x̂ , ũ).





J̄(0, x̂ , ũ) = E

N−1∑j=n

fj(X̄ (j), ũ(j , X̄ (j))

)+ g

(X̄ (N)



{fj(x , γ) +

∫Rd


}.







J̄(0, x̂ , ũ) = E

N−1∑j=n

fj(X̄ (j), ũ(j , X̄ (j))

)+ g

(X̄ (N)



{fj(x , γ) +

∫Rd


}.




Lower bound: representation due to [Rogers, 2007]

Assumption: there is a reference probability measure on Rd such that thetransition probabilities µn(x , γ) possess densities pn(x , ., γ) and there is areference density p∗n(x , .). Assumption satis�ed if di�usion coe�cient σ isuniformly elliptic.

Set φn(x , y , γ) :=pn(x,y ,γ)p∗n (x,y)

. Then, with X̄ (0) = x̂ , X̄ under P∗,

V̄ (x̂) = maxvE∗[

inf(γ0,...,γN−1)

{ N−1∏k=0

φk(X̄ (k), X̄ (k+1), γk

)· g(X̄ (N))

+N−1∑j=0

j−1∏k=0

φk(X̄ (k), X̄ (k+1), γk

) (fj(X̄ (j), γj) + ∆M(j)

)}],

where v ranges over all bounded {0, . . . ,N} × Rd → R with v(N, .) = g ,∆M(j) := v

(j+1, X̄ (j+1)

)φj

(X̄ (j), X̄ (j+1), γj

)−E∗

[v

(j+1, X̄ (j+1)

)φj

(X̄ (j), X̄ (j+1), γj

) ∣∣ X̄ (j)] .


Lower bound: representation due to [Rogers, 2007]

Assumption: there is a reference probability measure on Rd such that thetransition probabilities µn(x , γ) possess densities pn(x , ., γ) and there is areference density p∗n(x , .). Assumption satis�ed if di�usion coe�cient σ isuniformly elliptic.

Set φn(x , y , γ) :=pn(x,y ,γ)p∗n (x,y)

. Then, with X̄ (0) = x̂ , X̄ under P∗,

V̄ (x̂) = maxvE∗[

inf(γ0,...,γN−1)

{ N−1∏k=0

φk(X̄ (k), X̄ (k+1), γk

)· g(X̄ (N))

+N−1∑j=0

j−1∏k=0

φk(X̄ (k), X̄ (k+1), γk

) (fj(X̄ (j), γj) + ∆M(j)

)}],

where v ranges over all bounded {0, . . . ,N} × Rd → R with v(N, .) = g ,∆M(j) := v

(j+1, X̄ (j+1)

)φj

(X̄ (j), X̄ (j+1), γj

)−E∗

[v

(j+1, X̄ (j+1)

)φj

(X̄ (j), X̄ (j+1), γj

) ∣∣ X̄ (j)] .


Conclusions

1. Variant of approximate DP using Smolyak's algorithm.

2. Approximation scheme of �rst order almost as e�cient ashigher order schemes, but for a larger class of functions.

3. No useful a priori error bounds; Monte Carlo simulation toobtain a posteriori bounds.

4. To do: numerical robustness of the procedure, numericalexperiments!


Conclusions






Conclusions






Conclusions





Bibliography

Introduction


Smolyak's algorithm


Bibliography

Bibliography

Bibliography I

V. Barthelmann, E. Novak and K. Ritter.High dimensional polynomial interpolation on sparse grids.Adv. Comput. Math., 12: 273�288, 2000.

H. J. Bungartz and M. Griebel.Sparse grids.Acta Numerica, 13: 147-269, 2004.

M. Fischer and G. Nappo.Time discretisation and rate of convergence for the optimalcontrol of continuous-time stochastic systems with delay.Appl. Math. Optim., 57(2): 177�206, 2008.

Bibliography

Bibliography II

A. Klimke and B. Wohlmuth.Algorithm 847: spinterp: piecewise multilinear hierarchicalsparse grid interpolation in MATLAB.ACM Trans. Math. Softw., 31(4): 561-579, 2005.

N. V. Krylov.Approximating value functions for controlled degeneratedi�usion processes by using piece-wise constant policies.Electron. J. Probab., 4(Paper 2): 1-19, 1999.

R. Munos.Performance bounds in Lp norm for approximate valueiteration.SIAM Control Optim., 46, 541-561

Bibliography

Bibliography III

E. Novak.Deterministic and Stochastic Error Bounds in NumericalAnalysis.Lecture Notes in Mathematics 1349, Springer, New York, 1988.

L. C. G. Rogers.Pathwise stochastic optimal control.SIAM J. Control Optim., 46(3): 1116-1132, 2007.

A. Schreiber.Die Methode von Smolyak bei der multivariaten Interpolation.Dissertation, Georg-August-Universität zu Göttingen, 2000.

Bibliography

Bibliography IV

S. A. Smolyak.Quadrature and interpolation formulas for tensor products ofcertain classes of functions.Sov. Math., Dokl., 4: 240�243, 1963.

Bibliography

Thank you for your attention.

Approximation error

Let B(X ) := {f : X → R | f bounded} with norm ‖.‖∞.Information of cardinality n given by

In := {ξ : B(X ) → Rn | ξ(f ) = (f (x1), . . . , f (xn)), x i ∈ X}.

The set of algorithms which use information of cardinality n is

An := {Ψ = φ ◦ ξ | φ : Rn → B(X ), ξ ∈ In}.

Let F ⊂ B(X ). Approximation error of an algorithmus Ψ for F :

errApp(F ,Ψ) := supf ∈F

‖Ψ(f )− f ‖∞.

Approximation error based on information of cardinality n for F :

errApp(F , n) := infΨ∈An

errApp(F ,Ψ).

Approximation error




An := {Ψ = φ ◦ ξ | φ : Rn → B(X ), ξ ∈ In}.



‖Ψ(f )− f ‖∞.



errApp(F ,Ψ).

Approximation error




An := {Ψ = φ ◦ ξ | φ : Rn → B(X ), ξ ∈ In}.



‖Ψ(f )− f ‖∞.



errApp(F ,Ψ).

Approximation error




An := {Ψ = φ ◦ ξ | φ : Rn → B(X ), ξ ∈ In}.



‖Ψ(f )− f ‖∞.



errApp(F ,Ψ).

Smolyak's algorithm

Some properties of A(q, d) =∑|~i |≤q(∆

i1 ⊗ . . .⊗∆id ):

A(q, d) =∑

q−d+1≤|~i |≤q

(−1)q−|~i |(d − 1q − |~i |

) (U i1 ⊗ . . .⊗ U id

),

A(q, d) =q−1∑

k=d−1

(A(k , d−1)⊗∆q−k

),

A(q, d) = A(q−1, d) +∑

~i∈Nd ,|~i |=q

(∆i1 ⊗ . . .⊗∆id

).

A general result

[Smolyak, 1963], [Schreiber, 2000]:

Theorem

Let d ∈ N. For each i ∈ N, let U i : F i → V i be a continuous linearoperator with F i ⊂ F , V i ⊂ V Banach spaces. Let I1 : F → V belinear. Set Id := I1 ⊗ . . .⊗ I1.Suppose that, for each l ∈ {2, . . . , d}, the norms on ⊗lF und ⊗lVare uniformly compatible tensor norms. If there are constants B , c ,K such that ‖I1 − U i‖F ,V ≤ c B i , ‖I1‖F ,V ≤ K , then

‖Id −A(q, d)‖⊗dF ,⊗dV ≤ c Bq−d+1(

q

d−1

)max{K , c(1+B)}d−1.

Approximation error on sparse grids of Clenshaw-Curtis-type

Let n(q, d) := #H(q, d) be the number of points of the sparse gridin dimension d of degree q − d .

Assumption: the one-dimensional grids are nested and#X 1 = m1 = 1, #X

i = mi = 2i−1 + 1 se i > 1 (Clenshaw-Curtis).

If B = 2−k for some k ∈ N, then, under the assumptions of thetheorem and with n = n(q, d),

‖Id −A(q, d)‖⊗dF ,⊗dV≤ n−k(2d + log2(n))(1+k)(d−1)

c

(d−1)!k+1max{K , c(1+2−k)}d−1.

Approximation error on sparse grids of Clenshaw-Curtis-type

Let n(q, d) := #H(q, d) be the number of points of the sparse gridin dimension d of degree q − d .

Assumption: the one-dimensional grids are nested and#X 1 = m1 = 1, #X

i = mi = 2i−1 + 1 se i > 1 (Clenshaw-Curtis).

If B = 2−k for some k ∈ N, then, under the assumptions of thetheorem and with n = n(q, d),

‖Id −A(q, d)‖⊗dF ,⊗dV≤ n−k(2d + log2(n))(1+k)(d−1)

c

(d−1)!k+1max{K , c(1+2−k)}d−1.

IntroductionClassical control problems and approximate DPSmolyak's algorithmA posteriori error boundsBibliography

On the numerical solution of high-dimensional optimal control...

Documents

Transcript of On the numerical solution of high-dimensional optimal control...