Post on 09-May-2022
Markov Chain Specification
1 T o l z S is countable
2 Markov property3 P Xo R Ito r a C S
P Xt y I Xt a Ilse y
K y E S
ITC I should satisfy
1T x y e o D
IT K y I t ne S
YES
IT is a square matrix IT can be
yinfinite dimensional since Is I can be a
Example I Inventory system
demand periods u
mmmmI l l l l l
O
Stock replenishmentpointsn demand period numberXn inventory level at the begin of period n
G random discrete demand during period ns order levelu order upto level
1 Assume that G Gz one iidwith pm f f
2 Notice
XpEnn if s s xn e w
U Ent Xn E S
Example I Inventory system
demagd periods u
mmmmI l l l l l
O
Stock replenishmentpointss
Xn n e T is a Markov chain
with S u u t U z
T 0 1,2
and
Mocaif a u
0 OW
f x y se a e u
ITH'tfu y sees
Example III Epidemic Branching Chain
firmoffn generation number
1 v Xn number infectedIEEE IEEE in generation n
L v j
en reEEE Dv v
IEEE IEEE
S T O 1,2
Suppose each person in a generation independentlyinfects by others where pmf of G is f
Example III Epidemic Branching Chain
Md initial distribution of infected people1T 0,0 I IT 0 y O t y f O
IT r y PCG that then ywhere he Gz G are i i dwith pmf f K 31
Denote fu pmf of G test tha
O I 2
O I O O n n n
DI fill fill2 4 HDc
Example III Mutant Transmission
gene with d units med of whichare mutant
Bz Bz Bz Gen n
replicate3g BB Bae BB
I3h Bz Bz Bz Bz Gen ntl
Xn number of mutant units ingeneration n
S 0,1 d T 0 1,2
IT a y7 215 4 yeaead
0 otherwise
Ito n see o.is d given
Understand the Markov Property
Suppose Xt t e T is a
Markov chain Which of the
following are true statements
I P Xn me A X no X a Xrisen
I P X me A X an
II P Xn mE A XE Ao X EA XEAn.itn an
P Xn mEA Xn an
lI.P XnmEAXoEAo X EA glnEAn
P Xn mEA XnEAn
Why are Markov chains so useful
Let's calculate P X y IX a
IT a y P X y X a
IT ca y IT ca y
IT ca y P X y IX x X zz
P X zlXEP Xn y X z IT a z
z
IT cz.gg TCx.zZ
IT 2e Z 1T Cz yl
Why are Markov chains so useful
From 111 we see that
IT ca y The yIT Ca y IT x y
IT n'cry IT x y z
Since 2 holds fr any n y E S
we can write
IT'M IT 3
Don't confuse computational teeth
IT Cn y f CIT Ca yFH
So then
P X y
Hitting Time
Define
Tea min n o Xn EA
Ta is called the hitting time
of A Hitting time is first hittingkey Identity
time
P X y X a
nn m
P Try m 1T y yM 1
Why is this true
Hitting Time
P X y X a
n
P Try m Ty yM 1
Why is this true