Calculating principal eigenfunctions of non-negative...
Transcript of Calculating principal eigenfunctions of non-negative...
![Page 1: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/1.jpg)
Calculating principal eigenfunctions ofnon-negative integral kernels:
particle approximations and applications
Nikolas Kantas (Imperial)
COSMOS WORKSHOP PARIS, FEB. 2-5, 201616
joint work with Nick Whiteley (Bristol)
![Page 2: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/2.jpg)
Introduction
I On eigen-functions and related quantities of particular integraloperators (e.g. un-normalised Markov transition kernels)
I Applications:I rare event estimationI stochastic controlI statistical mechanics/physics (neutron transport, nuclear
fission, particle motion etc.)
I Outline:I Introduction on particle methods and simple Feynman-Kac
models in discrete timeI Eigen-quantities in linear algebra: finite state spaces and
matrices.I generalisations for measurable spaces and integral operatorsI particle approximations of eigen-quantitiesI rare events estimation example
![Page 3: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/3.jpg)
Introduction
I On eigen-functions and related quantities of particular integraloperators (e.g. un-normalised Markov transition kernels)
I Applications:I rare event estimationI stochastic controlI statistical mechanics/physics (neutron transport, nuclear
fission, particle motion etc.)
I Outline:I Introduction on particle methods and simple Feynman-Kac
models in discrete timeI Eigen-quantities in linear algebra: finite state spaces and
matrices.I generalisations for measurable spaces and integral operatorsI particle approximations of eigen-quantitiesI rare events estimation example
![Page 4: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/4.jpg)
Preliminaries
I Let M(x , dy) be a homogeneous Markov probability kernel on(X,B (X)) and let X0 = x .
IPotential function:
G = eU
where U : X ! R.I The non-negative kernel
Q (x , dy) := eU(x)M(x , dy)
defines a linear operator on functions
Q (') (x) :=
ˆQ (x , dy)' (y)
![Page 5: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/5.jpg)
Some Notation
Measures: say a measure µ on (X,B(X))
µ(') :=
ˆ'(x)µ(dx)
Integral kernels:I For K a (possibly un-normalised) Markov kernel on X ⇥ B (X),
K (') (x) :=
ˆK (x , dy)'(y)
andµK (·) :=
ˆµ(dx)K (x , ·)
I the n-fold iterate of kernel K : K n = K . . .K| {z }
n times
, K0 = Id
Think of kernel K being a matrix, µ a row vector and ' a vector!
![Page 6: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/6.jpg)
Feynman-Kac models
I Following notation/style of [Del Moral 04]I Usually in Monte Carlo we are interested to compute
something like:
�n,x(') = Qn (') (x) = E
x
"
exp
n�1X
k=0
U (Xk
)
!
' (Xn
)
#
I Simple example: particle motion in absorbing mediumI 1 � eU(x) probability of absorption at location xI M(x , dx 0) describe Markov dynamics of neutronI Qn
�
1�
(x) is probability of survival after n steps starting atX0 = x .
![Page 7: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/7.jpg)
Feynman-Kac models
I Feynman-Kac Models [Del Moral 04]:I more convenient to use a sequence of probability measuresI use the following sequence
⌘n
(') =Ex
h
exp⇣
P
n�1k=0 U (X
k
)⌘
' (Xn
)i
Ex
h
exp⇣
P
n�1k=0 U (X
k
)⌘i
I Update - prediction recursion:
⌘n+1(dy) =
⌘n
Q(x , dy)
⌘n
Q(1)
![Page 8: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/8.jpg)
Feynman-Kac models
Measure valued recursion :
⌘n+1 := �(⌘
n
) =⌘n
Q
⌘n
Q(1),
or in update - prediction steps
⌘n+1 = ⌘
n
M, ⌘n
=eU⌘
n
⌘n
(eU)
Product formula:
�n,x(') = ⌘
n
(')n�1Y
k=0
⌘k
(eU)
![Page 9: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/9.jpg)
A particle algorithm and some approximations
A simple particle algorithmI Initialization: sample i.i.d.
�
⇣ i0�
N
i=1 ⇠ µ,I For p = 1, ..., n: sample i.i.d.
�
⇣ ip
�
N
i=1
�
�
�
�
⇣ ip�1
�
N
i=1 ⇠N
X
j=1
exph
U⇣
⇣ jp�1
⌘i
P
N
j
0=1 exph
U⇣
⇣ j0
p�1
⌘iM⇣
⇣ jp�1, ·
⌘
Particle approximations, n � 0:
⌘Nn
:=1N
N
X
i=1
�⇣ in
, ⌘Nn
(') :=1N
N
X
i=1
'�
⇣ in
�
,
�Nn,x (') :=
n�1Y
k=0
⌘Nk
⇣
eU⌘
⌘Nn
(')
![Page 10: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/10.jpg)
The eigen problem
Under some regularity assumptions, Q has:I an isolated, real, maximal eigen-value �?,I a positive (right) eigen-function h?,I a positive (left) eigen-measure ⌘?I All-together
Q (h?) = �?h?, and ⌘?Q = �?⌘?, ⌘? (h?) = 1
I Do these quantities exist? Are they unique?I How can they be computed?I Is it possible to get extensions on
I Perron-Frobenious theoryI the power method from when Q, h?, ⌘? are matrix and vectors
resp.?
![Page 11: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/11.jpg)
The eigen problem - why do we care?Applications:
IStatistical physics
I h? could be viewed as the Schrödinger ground energy state formolecules,
I diffusion or quantum Monte CarloI [Rousset 06], [R. Assaraf, M. Caffarel, & A. Khelif 00],
[Makrini, B. Jourdain, and T. Lelièvre. 07].
I In optimal control:I discrete time problems with Kullback-Leibler divergence term
in the stage costI Q is as a multiplicative Bellman operator
I h? is a logarithmic transformation of the value function.
I[Albertini & Runggaldier 88], [Todorov 08]
I can be related to discretisations of certain continuous timemodels
I[Flemming 82], [Sheu 84], [Dai Pra, Meneghini,
Runggaldier,1996], [Kappen 05], [Theodorou 10]
![Page 12: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/12.jpg)
The eigen problem - why do we care?h? appears in large deviations theory of Markov chains;
I [Vere-Jones 67], [Ney & Nummelin 87], [Kontoyiannis & Meyn03]
I Let (Xn
; n � 0) is a Markov chain with transition kernel M,X0 = x , and G (x) := e↵U(x)
I Eigen-problem acts as multiplication Poisson eqn.
⇤(↵) = limn!1
1n
logEx
"
exp
↵n�1X
p=0
U(Xp
)
!#
,
h?(x) = limn!1
Ex
"
exp
↵n�1X
p=0
U(Xp
)� n⇤(↵)
!#
I Let ⇤0(↵) = c , �2↵ = ⇤00(↵) and some conditions on U
Px
"
n�1X
p=0
U(Xp
) > nc
#
⇠ h?(x)
↵p
2⇡�2↵
exp (�n⇤⇤(c))
![Page 13: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/13.jpg)
The eigen problem: the “twisted” Markov kernel
I A related object of interest:
P?(x , dx0) :=
Q(x , dx 0)h?(x 0)h?(x)�?
. (1)
I Many applications of interest :I estimation of tail probabilities of Markov chains:
Icertain P? is optimal changes of measure
[Bucklew, Ney, Sadowski 90], [Dupuis & Wang 05]
I optimal control:I P? is optimally controlled Markov transition kernel.
I particle motion in absorbing media:I P? is the Markov transition kernel of a particle conditional on
long-term survival
[Del Moral, Miclo 03], [Del Moral & Doucet 04], [Rousset 06]
I branching processes: [Harris 51], [Athreya 00]
![Page 14: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/14.jpg)
Eigen -values, -measures, -functions of Q�
= eUM�
We look to compute quantities that satisfy:
Q (h?) = �?h?, and ⌘?Q = �?⌘?, ⌘? (h?) = 1
I In the finite state space caseI When do these quantities exist? Are they unique?I How can they be computed?
![Page 15: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/15.jpg)
Basic Perron-Frobenious theory
Let Q be a square n ⇥ n matrix with real positive entries. Then:
1. there is one real and isolated eigenvalue �? on the spectralradius and for any other eigenvalue �, |�| < �?.
2. Perron vectors: for �? there exist unique positive right and lefteigenvectors h?, ⌘? of size n ⇥ 1 and 1 ⇥ n resp. that satisfy
Qh? = �?h?, ⌘?Q = �?⌘?, ⌘?h? = 1
3. We have:��n
? Qn ! h?⌘?
![Page 16: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/16.jpg)
How to compute �?, h?, ⌘??
I The power method: hk
! h?,
hk+1 =
Qhk
kQhk
k
assuming h00h? 6= 0. Similar for ⌘?.I Approximation for the eigenvalue �
k
! �?, e.g. using Rayleighquotient:
�k
= arg min�
kQhk
� �hk
k2 =h⇤k
Qhk
h⇤k
hk
I Gelfand’s formula9Qk91/k ! �?
![Page 17: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/17.jpg)
General State Space: regularity conditions for Q
Recall Q = eUM, M a Markov probability kernel, U a real functionon X
Regularity condition (A1): there exists a probability measure ⌫ suchthat
dQ(x , ·)d⌫
(y) = q(x , y)
and0 < ✏� q(x , y) ✏+ < 1
I Under (A1) Q is aperiodic and irreducible.I A bit restrictive but hopefully this can be relaxed, e.g.
[Kontoyiannis and Meyn 03] etc.
![Page 18: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/18.jpg)
Perron-Frobenius Theory for Q
I [...,Nummelin 84] Under (A1) there exist: a maximal andisolated eigenvalue �?, a positive eigen-function, probabilityeigen-measure:
Q (h?) = �?h?, and ⌘?Q = �?⌘?, ⌘? (h?) = 1
and these are unique.
I Multiplicative Ergodic Theorem: For any n � 1,
supx2X
sup|'|<1
�
���n
? Qn (') (x)� h?(x)⌘?(')�
� 2✓
✏+
✏�
◆2
⇢n
or
9��n
? Qn
? � h? ⌦ ⌘?9 2⇢n✓
✏+
✏�
◆2
where ⇢ = 1 � (✏�/✏+).
![Page 19: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/19.jpg)
Perron-Frobenius Theory for Q
I [...,Nummelin 84] Under (A1) there exist: a maximal andisolated eigenvalue �?, a positive eigen-function, probabilityeigen-measure:
Q (h?) = �?h?, and ⌘?Q = �?⌘?, ⌘? (h?) = 1
and these are unique.I Multiplicative Ergodic Theorem: For any n � 1,
supx2X
sup|'|<1
�
���n
? Qn (') (x)� h?(x)⌘?(')�
� 2✓
✏+
✏�
◆2
⇢n
or
9��n
? Qn
? � h? ⌦ ⌘?9 2⇢n✓
✏+
✏�
◆2
where ⇢ = 1 � (✏�/✏+).
![Page 20: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/20.jpg)
The twisted probability kernel (Doob’s h kernel)
I The MET is derived using the properties of the twisted Markovkernel
P?(x , dy) :=Q(x , dy)h?(y)
h?(x)�?. (2)
I P? is ergodic with a unique invariant probability distribution,denoted by ⇡? and for all n � 1,
supx2X
supA2B(X)
|Pn
? (x ,A)� ⇡?(A)| 2⇢n (3)
d⇡?/d⌘? = h? (4)
or9Pn
? � 1 ⌦ ⇡?9 2⇢n
where ⇢ = 1 � (✏�/✏+).
![Page 21: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/21.jpg)
How to approximate ⌘?,�??I [Del Moral, Miclo 02], [Del Moral, Doucet 04], [Rousset 06]I Consider the forward recursion for n � 0:
⌘n+1 = �(⌘
n
) =⌘n
Q
⌘n
Q(1),
�n
= ⌘n
Q(1) = ⌘n
(eU)
I Recall
⌘?Q = �?⌘? ) ⌘?Q(1) = ⌘?(eU) = �?.
) � (⌘?) = ⌘?
I In fact, for any initial condition ⌘0
k⌘n+1 � ⌘?k C⇢n
|�n
� �?| C 0⇢n
![Page 22: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/22.jpg)
How to approximate h?, P??
I Often this is enoughI when M is µ- reversible then h?
µ(h?)= d⌘?
dµI [Rousset 06], [Makrini, B. Jourdain, and T. Lelièvre. 07].
I When this does not apply but still within (A1)I Keep previous forward sequence (⌘
p
,�p
)p�0
I Find a sequence of functions (hp
)p�0 such that
Q (hp+1) = �
p
hp
, ⌘p
Q = �p+1⌘p+1, ⌘
p+1(hp+1) = 1
![Page 23: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/23.jpg)
How to approximate h?, P??
I Often this is enoughI when M is µ- reversible then h?
µ(h?)= d⌘?
dµI [Rousset 06], [Makrini, B. Jourdain, and T. Lelièvre. 07].
I When this does not apply but still within (A1)I Keep previous forward sequence (⌘
p
,�p
)p�0
I Find a sequence of functions (hp
)p�0 such that
Q (hp+1) = �
p
hp
, ⌘p
Q = �p+1⌘p+1, ⌘
p+1(hp+1) = 1
![Page 24: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/24.jpg)
How to approximate h?, P??I Backward recursionI Set h
n,n(x) := 1, for 0 p < n.
hp,n(x) :=
Q (hp+1,n)
�p
=Q(n�p)(1)(x)Q
n�1`=p
�`,
andPp,n(x , dx
0) :=Q(x , dx 0)h
p,n
�p�1hp�1
I We can invoke the MET
9��n
? Q(n)? � h? ⌦ ⌘?9 2⇢n
✓
✏+
✏�
◆2
to deduce
khp,n � h?k C⇢p^(n�p)
9Pp,n � P?9 C 0⇢p^(n�p)
![Page 25: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/25.jpg)
How to approximate h?, P??I Backward recursionI Set h
n,n(x) := 1, for 0 p < n.
hp,n(x) :=
Q (hp+1,n)
�p
=Q(n�p)(1)(x)Q
n�1`=p
�`,
andPp,n(x , dx
0) :=Q(x , dx 0)h
p,n
�p�1hp�1
I We can invoke the MET
9��n
? Q(n)? � h? ⌦ ⌘?9 2⇢n
✓
✏+
✏�
◆2
to deduce
khp,n � h?k C⇢p^(n�p)
9Pp,n � P?9 C 0⇢p^(n�p)
![Page 26: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/26.jpg)
An ideal algorithm like the power method
ForwardInitialization: Set ⌘0 = µ,For p = 1, ..., 2n, :
Set ⌘p
= ⌘n
Q
�n
, �n
= ⌘n
Q(1)Backward
Initialization: Set h2n,2n(x) = 1, x 2 XFor p = 2n � 1, ..., n, :
Set hp,2n(x) = (�
p
)�1 Q (hp+1,2n) (x), x 2 X
Set Pp+1,n(x , dx 0) :=
Q(x ,dx 0)hp+1,n(x 0)
�p
h
p
(x) , x , x 0 2 X
khn,2n � h?k C⇢n
9Pp,n � P?9 C 0⇢n
![Page 27: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/27.jpg)
A particle algorithm
I Forward
I Initialization: sample i.i.d.�
⇣ i0�
N
i=1 ⇠ µ,I For p = 1, ..., 2n: sample i.i.d.
�
⇣ ip
�
N
i=1
�
�
�
�
⇣ ip�1
�
N
i=1 ⇠N
X
j=1
exph
U⇣
⇣ jp�1
⌘i
P
N
j
0=1 exph
U⇣
⇣ j0
p�1
⌘iM⇣
⇣ jp�1, ·
⌘
,
I BackwardI Initialization: set h2n,2n(x) = 1, x 2 XI For p = 2n � 1, ..., n, set
hNp,2n(x) =
N
X
j=1
q(x , ⇣ jp+1)
P
N
i=1 q(⇣i
p
, ⇣ jp+1)
hNp+1,2n(⇣
j
p+1), x 2 X
![Page 28: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/28.jpg)
A particle algorithm
I Forward
I Initialization: sample i.i.d.�
⇣ i0�
N
i=1 ⇠ µ,I For p = 1, ..., 2n: sample i.i.d.
�
⇣ ip
�
N
i=1
�
�
�
�
⇣ ip�1
�
N
i=1 ⇠N
X
j=1
exph
U⇣
⇣ jp�1
⌘i
P
N
j
0=1 exph
U⇣
⇣ j0
p�1
⌘iM⇣
⇣ jp�1, ·
⌘
,
I BackwardI Initialization: set h2n,2n(x) = 1, x 2 XI For p = 2n � 1, ..., n, set
hNp,2n(x) =
N
X
j=1
q(x , ⇣ jp+1)
P
N
i=1 q(⇣i
p
, ⇣ jp+1)
hNp+1,2n(⇣
j
p+1), x 2 X
![Page 29: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/29.jpg)
Particle Approximations
I (Forward pass) Target eigen-measure and eigen-value
⌘Nn
:=1N
N
X
i=1
�⇣ in
, ⌘Nn
(') :=1N
N
X
i=1
'�
⇣ in
�
, �Nn
:= ⌘Nk
⇣
eU⌘
I (Backward pass) eigen-function
hNn,2n(x) =
N
X
j=1
q(x , ⇣ jn+1)
P
N
i=1 q(⇣i
n
, ⇣ jn+1)
hNn+1,2n(⇣
j
n+1)
I (Backward pass) twisted kernel:
PN
n,2n(x , dx0) :=
1hNn�1,2n(x)
N
X
j=1
q(x , ⇣ jn
)P
N
i=1 q(⇣i
n�1, ⇣j
n
)hNn,2n(⇣
j
n
)�⇣ jn
�
dx 0�
.
![Page 30: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/30.jpg)
Particle Approximations
I (Forward pass) Target eigen-measure and eigen-value
⌘Nn
:=1N
N
X
i=1
�⇣ in
, ⌘Nn
(') :=1N
N
X
i=1
'�
⇣ in
�
, �Nn
:= ⌘Nk
⇣
eU⌘
I (Backward pass) eigen-function
hNn,2n(x) =
N
X
j=1
q(x , ⇣ jn+1)
P
N
i=1 q(⇣i
n
, ⇣ jn+1)
hNn+1,2n(⇣
j
n+1)
I (Backward pass) twisted kernel:
PN
n,2n(x , dx0) :=
1hNn�1,2n(x)
N
X
j=1
q(x , ⇣ jn
)P
N
i=1 q(⇣i
n�1, ⇣j
n
)hNn,2n(⇣
j
n
)�⇣ jn
�
dx 0�
.
![Page 31: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/31.jpg)
Lr error estimates
Under (A1) we have 8n � 1,N � 1,r � 1 there exist constantsB(r),C such that:
supx2X
EN
h
�
�
�
hNn,2n(x)� h?(x)
�
�
�
r
i1/r B
h
(r)pN
+ Ch
⇢n (5)
supx2X
supA2B(X)
EN
h
�
�
�
PN
n,2n (x ,A)� P? (x ,A)�
�
�
r
i1/r B
P
(r)pN
+ CP
⇢n,
(6)where ⇢ = (1 � (✏�/✏+)).
In part appeal to earlier results for particle smoothers [Del Moral,Doucet, Singh 10] and [Douc, Garivier, Moulines & Olsson 11]
![Page 32: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/32.jpg)
An application on rare events estimation
I Let (Xn
; n � 0) be a Markov chain with transition Minitialized from some X0 = x .
I Also let U : X ! [�1, 1]I Our objective is for some � 2 (0, 1) and m � 1, to estimate
the deviation probability
⇡m
(�) := Px
0
@
m
X
p=1
U (Xp
) > m�
1
A . (7)
where Px
is the law of the chain.
![Page 33: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/33.jpg)
Sequential Importance Sampling (IS) for rare events
I Let C be the collection of Markov transitions M: there exist0 < ✏�, ✏+ < 1 and a probability measure ⌫ such that ⌫ ⌧ ⌫
(C) ⌫ (·) ✏� M(x , ·) ✏+⌫ (·) , 8x ,ˆ ✓
d⌫
d ⌫(x)
◆2⌫ (dx) < 1,
I The sequential importance sampling (IS) estimator of ⇡m
(�) is
b⇡m
(�, L) :=1L
L
X
i=1
I
2
4
m
X
p=1
U�
X i
p
�
> m�
3
5
dPx
dPx
�
X i
0, ...,Xi
m
�
,
(8)where
��
X i
0,Xi
1, ...,�
; i = 1, ..., L
are L iid Markov chains,each with transition M and law denoted by P
x
![Page 34: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/34.jpg)
Sequential Importance Sampling (IS) for rare events
I Let C be the collection of Markov transitions M: there exist0 < ✏�, ✏+ < 1 and a probability measure ⌫ such that ⌫ ⌧ ⌫
(C) ⌫ (·) ✏� M(x , ·) ✏+⌫ (·) , 8x ,ˆ ✓
d⌫
d ⌫(x)
◆2⌫ (dx) < 1,
I The sequential importance sampling (IS) estimator of ⇡m
(�) is
b⇡m
(�, L) :=1L
L
X
i=1
I
2
4
m
X
p=1
U�
X i
p
�
> m�
3
5
dPx
dPx
�
X i
0, ...,Xi
m
�
,
(8)where
��
X i
0,Xi
1, ...,�
; i = 1, ..., L
are L iid Markov chains,each with transition M and law denoted by P
x
![Page 35: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/35.jpg)
Importance Sampling for rare events: twisting the kernelI AIM: the relative variance of IS
Ex
"
✓
b⇡m
(�, L)
⇡m
(�)� 1
◆2#
=1L
0
@
Ex
h
b⇡m
(�, 1)2i
⇡m
(�)2� 1
1
A (9)
not to grow exponentially fast in m.
I Consider
G↵0(x) := e↵0U(x), Q↵0
�
x , dx 0�
:= G↵0(x)M�
x , dx 0�
and
⇤?(↵) := limn!1
1n
logEx
2
4exp
0
@↵n�1X
p=0
U(Xp
)
1
A
3
5 , ↵ 2 R
I (t) := sup↵2R
[t↵� ⇤?(↵)] , t 2 R
![Page 36: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/36.jpg)
Importance Sampling for rare events: twisting the kernelI AIM: the relative variance of IS
Ex
"
✓
b⇡m
(�, L)
⇡m
(�)� 1
◆2#
=1L
0
@
Ex
h
b⇡m
(�, 1)2i
⇡m
(�)2� 1
1
A (9)
not to grow exponentially fast in m.I Consider
G↵0(x) := e↵0U(x), Q↵0
�
x , dx 0�
:= G↵0(x)M�
x , dx 0�
and
⇤?(↵) := limn!1
1n
logEx
2
4exp
0
@↵n�1X
p=0
U(Xp
)
1
A
3
5 , ↵ 2 R
I (t) := sup↵2R
[t↵� ⇤?(↵)] , t 2 R
![Page 37: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/37.jpg)
Importance Sampling with large deviations Bucklew et al. 901. I (t) is a non-negative, strictly convex function with I (t) = 0 if
and only if t = ⇤0?(0).2. For any � 2 (0, 1), the following large deviation principle holds
limm!1
1m
log ⇡m
(�) = � inft2[�,1)
I (t).
3. For any � 2 (0, 1) and M in C, the importance samplingestimator satisfies
limm!1
1m
logEx
h
b⇡m
(�, 1)2i
� �2 inft2[�,1)
I (t). (10)
4. For any � 2 (0, 1) and ↵ the unique solution of
⇤0? (↵) = �
the twisted kernel P↵? is the unique member of C for which
equality holds (asymptotically efficient.)
![Page 38: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/38.jpg)
Importance Sampling for rare events using lack of bias
I Let�
Xp
; p = 0, ...,m�
be a non-homogeneous Markov chainwith transitions
X0 = x , Xp
⇠ PN
↵,n+p,2n�
Xp�1, ·
�
, p � 1.
I Then we have the following lack of bias property:
EN
2
4I
2
4
m
X
p=1
U�
Xp
�
> m�
3
5
m�1Y
p=0
�N↵,n+p
G↵�
Xp
�
hN↵,n,2n�
X0�
hN↵,n+m,2n�
Xm
�
3
5 = ⇡m
(�),
EN,↵ for expectation w.r.t. the joint law of
�
Xp
�
and the particlesystem.
![Page 39: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/39.jpg)
Numerical example
Take X = [�c , c] and consider an ergodic Gaussian transitionkernel with support restricted to [�c , c],
M(x , dy) =exp
⇣
�12�
y � x
2�2⌘
⇣
1 � erf⇣
c+x/2p2
⌘⌘p2⇡
I[�c,c](y)dy ,
Consider U defined by
U (x) =
8
>
<
>
:
�1 x �1x x 2 (�1, 1)1 x � 1.
![Page 40: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/40.jpg)
Numerical example
!6 !4 !2 0 2 4 6!8
!7
!6
!5
!4
!3
!2
!1
0
1
2
!10 !8 !6 !4 !2 0 2 4 6 8 10!1
!0.8
!0.6
!0.4
!0.2
0
0.2
0.4
0.6
0.8
1
Figure : Left: each of the solid curves shows an approximation of
[↵t � ⇤?(↵)] against ↵, with each curve corresponding to a different
value of t in the range [�0.8, 0.8]. The cross on each curve indicates
its maximum and thus approximates sup↵ [↵t � ⇤?(↵)] = I (t). Right:
⇤0?(↵) against ↵ approximated using finite differences.
![Page 41: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/41.jpg)
Numerical example
0 5 10 15 20 25 30 35 40 45 5010
!16
10!14
10!12
10!10
10!8
10!6
10!4
10!2
100
0 5 10 15 20 25 30 35 40 45 500
10
20
30
40
50
60
70
80
90
100
Figure : Left: estimated value of ⇡m
(�) against m, for: �,� = 0.8;⇤,� = 0.9, and +, � = 0.99. Right: solid lines show sample relativevariance of the estimated value of ⇡
m
(0.9) against m for: �, ↵ = 1;+,↵ = 2; ⇤, ↵ = 4; ⇤, ↵ = 8; and ⇥,↵ = 16. Dashed line shows samplerelative variance of b⇡
m
(0.9, 1) in the case M = M.
![Page 42: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/42.jpg)
Conclusions/Extensions
I Some interesting examples/extensions on stochastic control:I h
p
, h? are value functions for particular finite/ infinite horizonproblems resp.
I In the rare events example, can use also other ideas from SMCor adaptive IS [Dupuis and Wang 05] to improve things.
I When hp
(x), h?(x) are the inference objectiveI there is some interest in variance reduction
I This is a batch scheme with cost O(N2n)
I can this be reduced in some way?
![Page 43: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/43.jpg)
ReferencesJ.A. Bucklew, P. Ney, J.S. Sadowsky (1990) Monte Carlo simulations and largedeviations theory for uniformly recurrent Markov chains J. Appl. Prob.
P. Del Moral and L. Miclo. (2003) Particle approximations of Lyapunovexponents connected to Schrodinger operators and Feynman-Kac semigroups.ESAIM Probab. Stat.
P. Del Moral, A. Doucet, and S.S. Singh. (2011) A backward particleinterpretation of Feynman-Kac formulae. ESAIM Math. Model. Numer. Anal.
R. Douc, Garivier, A. and Moulines, E. and Olsson, J. (2011) Sequential MonteCarlo smoothing for general state space hidden Markov models. Ann. Appl.Probab..
E. Nummelin. (1984) General irreducible Markov chains and non-negativeoperators. CUP.
I. Kontoyiannis and S. P. Meyn (2003) Spectral theory and limit theorems forgeometrically ergodic Markov processes,Ann. Appl. Probab. 13 (1), 304-362.
N. Whiteley and N. Kantas (2012). A particle method for approximatingprincipal eigen-functions and related quantities. Submitted.
N. Whiteley, N. Kantas and A. Jasra, (2012) Linear Variance Bounds for ParticleApproximations of Time-Homogeneous Feynman-Kac Formulae, SPA.
N. Whiteley (2013) Stability properties of some particle filters. Ann. Appl.Probab...
![Page 44: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/44.jpg)
Appendix: relaxing (A1)
I Similar to [Whiteley, N.K., Jasra 12], [Whiteley 13]I Use assumptions from [Kontoyiannis & Meyn 03]:
I using multiplicative drift conditions , i.e. there exists bd
< 1and Lyapunov function V such that:
Q�
eV�
eV (1��)+b
d
IC
d .
I a stronger MET is derived for QI can be verified in general for following cases:
1. bounded functions U and non-ergodic kernels M2. unbounded above functions U and multiplicative ergodic
kernels M
I Can be verified in practice for realistic examples
![Page 45: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/45.jpg)
Optimal Control
Let (Xn
; n � 0) be a controlled Markov chain initialized fromX0 = x and X
n
⇠ M f
n�1(Xn�1, ·)
V0(x) = inff 2Hn
Ef
x ,0
2
4
n�1X
p=0
⇣
U(Xp
) +KL⇣
M f
p
�
�
�
M⌘
(Xp
)⌘
+⌦(Xn
)
3
5 ,
V?(x) = inff 2HN
lim supn!1
1nEf
x ,0
2
4
n
X
p=0
⇣
U(Xp
) +KL⇣
M f
p
�
�
�
M⌘
(Xp
)⌘
3
5
![Page 46: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/46.jpg)
Finite horizon setupI Let V
n
= ⌦, Q = e�UM, �p
= e�⇤p and V
p+1 = � log hp
thebackward recursion
Q (hn+1) = �
n
hn
corresponds to Bellman eqn.
Vp
(x) = U(x)�⇤p
+ inff
p
2H
n
KL⇣
M f
p
�
�
�
M⌘
(x) +M f
p (Vp+1) (x)
o
I For p = 1, . . . , nI Run forward particle system to compute particle
approximations for (⌘p
),
I For p = n, n � 1 . . . , 1I Run backwards in time to compute particle approximations for
hp
and Pp
![Page 47: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/47.jpg)
Infinite horizon problem: particle value iteration
I Similarly, V?(x) = � log h?(x), &? = � log �?
Q (h?) = �?h?
is equivalent to a solution (V?,⇤?) of the average-costoptimality equation:
V?(x) + ⇤? = infh2H
h
U(x) + KL⇣
Mh
�
�
�
M⌘
(x) + Mh (V?) (x)i
.
I Use previous algorithm for n very largeI forward particle system up to time 2n to compute particle
approximations (⌘Np
),I then backwards to time n to compute particle approximations
hNn,2n and PN
n,2n
![Page 48: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/48.jpg)
Example: Gaussian model
Consider the controlled dynamics:
Xn
=
X 1n
V 1n
�
=
1 ⌧0 1
�
Xn�1 +
⌧ ⌧2
0 ⌧
�
(Wn
+ An
)
consider the state-dependent-only part of the stage cost:
U(x) / (1 � I(��,�)(x1))
which penalises states outside (��, �).
![Page 49: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/49.jpg)
Example: Gaussian model
!1 !0.5 0 0.5 1!1
0
1
2
3
4
5
x
C(x
)
!1 !0.5 0 0.5 1!1
0
1
2
3
4
5
x
V19(x
)
!1 !0.5 0 0.5 1!1
0
1
2
3
4
5
x
V15(x
)
!1 !0.5 0 0.5 1!1
0
1
2
3
4
5
x
V10(x
)
Figure : Estimated horizon average cost optimal value function Vn
(x)against x for various n. Here T = 20,
T
= C , ⌧ = 1,N = 500
![Page 50: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/50.jpg)
Example: Cox-Ingersoll-Ross process
I Control free model for M is Euler discretisation of CIR
dXt
= ✓ (µ� Xt
) dt + �p
Xt
dWt
,
where {Wt
} is standard 1-D Brownian motion, ✓ > 0 is thereversion rate, µ > 0 is the level of mean reversion and � > 0specifies the volatility.
I Stage cost specified by
U(x) = 2I[0,10��](x) + I[10+�,1)(x), (11)
which penalises states outside (10 � �, 10 + �).
![Page 51: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/51.jpg)
Example: Cox-Ingersoll-Ross process
4 6 8 10 12 14 16 18 20
0
2
4
6
8
10
12
14
16
18
20
Figure : Estimated infinite-horizon average cost optimal value functionV?(x) against x for: �, � = 5;⇥, � = 4;⇤, � = 3; +, � = 2.
![Page 52: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/52.jpg)
Appendix : connection to some pde’s
Numerical solutions for some class of parabolic PDEs for t 2 [0,T ]
(v)t
+ a (v)x
+12�2 (v)
xx
+ 'v = 0, v(·,T ) = (·)
using Q(n�p)( )(x) =
hp,n(x)
n�1Y
l=p
�l
⇡ Etx
exp✓ˆ
T
t
U(Xs
)ds
◆
'(XT
)
�
= v(x , t)
where the expectation is taken conditional to Xt
= x w.r.t.
dXt
= a(Xt
)dt + �(Xt
)dWt
.
![Page 53: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/53.jpg)
Appendix: a stochastic control exampleI For the following controlled Markov chain
dXt
= (a(Xt
) + B(Xt
)At
) dt + �(Xt
)dWt
,
let the total cost or value function for t � 0:
V (x , t) = infU2L2(·),s2(t,T )
Etx
ˆT
t
L(Xs
,As
)ds + (XT
)
�
,
with the stage cost being:
L(x ,A) = C (x) +12A
0R(x)A,
I IF we have BRB 0 = ��0 then the Hamilton-Jacobi-Bellmanequation implies
V (x , t) = � log v(x , t), C = �U, = � log' (12)
with v(x , t) as in the previous slide and for the optimal control:
A⇤(x , t) = R(x)�1B(x)0rx
v(x , t).
![Page 54: Calculating principal eigenfunctions of non-negative …cermics.enpc.fr/~stoltz/COSMOS_conference/kantas.pdfCalculating principal eigenfunctions of non-negative integral kernels: particle](https://reader034.fdocuments.us/reader034/viewer/2022050219/5f646db5258dc735a00c0608/html5/thumbnails/54.jpg)
Appendix: a stochastic control exampleI For the following controlled Markov chain
dXt
= (a(Xt
) + B(Xt
)At
) dt + �(Xt
)dWt
,
let the total cost or value function for t � 0:
V (x , t) = infU2L2(·),s2(t,T )
Etx
ˆT
t
L(Xs
,As
)ds + (XT
)
�
,
with the stage cost being:
L(x ,A) = C (x) +12A
0R(x)A,
I IF we have BRB 0 = ��0 then the Hamilton-Jacobi-Bellmanequation implies
V (x , t) = � log v(x , t), C = �U, = � log' (12)
with v(x , t) as in the previous slide and for the optimal control:
A⇤(x , t) = R(x)�1B(x)0rx
v(x , t).