Post on 23-Dec-2015
1
Introduction to Stochastic ModelsIntroduction to Stochastic ModelsGSLM 54100GSLM 54100
2
OutlineOutline
discrete-time Markov chain motivation
example
transient behavior
3
MotivationMotivation
What happens if Xn’s are dependent?
many dependent systems, e.g., inventory across periods
state of a machine
customers unserved in a distribution system
time
excellent
good
fair
bad
4
MotivationMotivation
any nice limiting results for dependent Xn’s?
no such result for general dependent Xn’s
nice results when Xn’s form a discrete-time
Markov Chain
1 ???
N
nNn
X
N
{ }11
???n
N
X sNn
N
5
Discrete-Time, Discrete-State Discrete-Time, Discrete-State Stochastic ProcessStochastic Process
a stochastic process: a sequence of indexed random variables, e.g., {Xn}, {X(t)}
a discrete-time stochastic process: {Xn}
a discrete-state stochastic process, e.g., state {excellent, good, fair, bad}
set of states {e, g, f, b} {1, 2, 3, 4} {0, 1, 2, 3}
state to describe weather {windy, rainy, cloudy, sunny}
6
Markov PropertyMarkov Property
a discrete-time, discrete-state stochastic process possesses the Markov property if P{Xn+1 = j|Xn = i, Xn−1 = in−1, . . . , X1 = i1, X0 = i0} = pij,
for all i0, i1, …, in1, in, i, j, n 0
time frame: presence n, future n+1, past {i0, i1, …, in1}
meaning of the statement: given presence, the past and the future are conditionally independent
the past and the future are certainly dependent
7
One-Step Transition Probability MatrixOne-Step Transition Probability Matrix
pij 0, i, j 0,
00 01 02
10 11 12
0 1 2
...
...
i i i
p p p
p p p
p p p
P M M M
K
M M M
01, 0,1, 2,...ij
jp i
8
Example 4-1 Example 4-1 Forecasting the WeatherForecasting the Weather
state {rain, not rain}
dynamics of the system rains today rains tomorrow w.p. does not rain today rains tomorrow w.p.
weather of the system across the days, {Xn}
1
1
P
9
Example 4-2 Example 4-2 A Communication SystemA Communication System
digital signals in 0 and 1 a signal remaining unchanged with probability
p on passing through a stage, independent of everything else
state = value of the signal {0, 1}
Xn: value of the signal before entering the nth stage 1
1
p p
p p
P
10
Example 4-3 Example 4-3 The Mood of a PersonThe Mood of a Person
mood {cheerful (C), so-so (S), or glum (G)} cheerful today C, S, or G tomorrow w.p. 0.5, 0.4, 0.1 so-so today C, S, or G tomorrow w.p. 0.3, 0.4, 0.3 glum today C, S, or G tomorrow w.p. 0.2, 0.3, 0.5
Xn: mood on the nth day, such that mood {C, S, G}
{Xn}: a 3-state Markov chain (state 0 = C, state 1 = S, state 2 = G) 0.5 0.4 0.1
0.3 0.4 0.3
0.2 0.3 0.5
P
11
Example 4.4Example 4.4Transforming a Process into a Transforming a Process into a DTMCDTMC
raining or not today depending on the weather conditions of the last two days rained for the past two days will rain tomorrow
w.p. 0.7 rained today but not yesterday will rain
tomorrow w.p. 0.5 rained yesterday but not today will rain
tomorrow w.p. 0.4 not rained in the past two days will rain
tomorrow w.p. 0.2
12
Example 4.4Example 4.4Transforming a Process into a Transforming a Process into a DTMCDTMC
state 0 if it rained both today and yesterday 1 if it rained today but not yesterday 2 if it rained yesterday but not today 3 if it did not rain either yesterday or today
0.7 0 0.3 0
0.5 0 0.5 0
0 0.4 0 0.6
0 0.2 0 0.8
P
13
Example 4.5Example 4.5A Random Walk ModelA Random Walk Model
a discrete-time Markov chain of number of states {…, -2, -1, 0, 1, 2, …}
random walk: for 0 < p < 1,
pi,i+1 = p = 1 − pi,i−1, i = 0, 1, . . .
14
Example 4.6Example 4.6A Gambling ModelA Gambling Model
each play of a game a gambler gaining $1 w.p. p, and losing $1 o.w.
end of the game: a gambler either broken or accumulating $N transition probabilities:
pi,i+1 = p = 1 − pi,i−1, i = 1, 2, . . . , N − 1; p00 = pNN = 1 example for N = 4
state: Xn, the gambler’s fortune after the n play {0, 1, 2, 3, 4}
1 0 0 0 0
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 0 1
p p
p p
p p
P
15
Example 4.7Example 4.7
insurance premium paid on a year depending on the number of claims made last year
16
Example 4.7Example 4.7
# of claims in a year ~ Poisson()
, 0!
k
ka e kk
0 1 2
0 1
0
0
1
0 1
0 0 1
0 0 1
a a a
a a
a
a
P
17
Transient Behavior Transient Behavior {Xn} for weather condition
0 if it rained both today and yesterday 1 if it rained today but not yesterday 2 if it rained yesterday but not today 3 if it did not rain either yesterday or today
suppose yesterday rained and today does not, what is the weather forecast for tomorrow? for 10 days from now?
0.7 0 0.3 0
0.5 0 0.5 0
0 0.4 0 0.6
0 0.2 0 0.8
P
18
mm-Step Transition Probability Matrix-Step Transition Probability Matrix
one-step transition probability matrix, P = [pij], where pij = P(X1 = j|X0 = i)
m-step transition probability matrix where
claim: P(m) = Pm
( )( ) ,mmijp P
( )0( | )m
mijp P X j X i
19
mm-Step Transition Probability Matrix-Step Transition Probability Matrix
Markov chain {Xn} for weather
Xn {r, c, s}, where r = rainy, c = cloudy, s = sunny
0.5 0.4 0.1
0.3 0.4 0.3
0.2 0.3 0.5
P
n = 0 n = 1 n = 2
State = r
State = c
State = s
pcr
pcc
pcs
prr
pcr
psr
(2)cr rr cc cr cs srijp p p p p p p
20
mm-Step Transition Probability Matrix-Step Transition Probability Matrix(2)
2 0
2 1 0 2 1 0
2 1 0
2 1 0 1 0
2 1 0 1 0
( | )
= ( , | ) ( , | )
+ ( , | )
= ( | , ) ( | )
+ ( | , ) ( | )
crp P X r X c
P X r X r X c P X r X c X c
P X r X s X c
P X r X r X c P X r X c
P X r X c X c P X c X c
2 1 0 1 0
2 1 1 0
2 1 1 0
2 1 1 0
1 0 1 0
+ ( | , ) ( | )
= ( | ) ( | )
+ ( | ) ( | )
+ ( | ) ( | )
= ( | ) ( |
P X r X s X c P X s X c
P X r X r P X r X c
P X r X c P X c X c
P X r X s P X s X c
P X r X r P X r X c
1 0 1 0
1 0 1 0
)+ ( | ) ( | )
+ ( | ) ( | )
=
= rr cr cr cc sr cs
cr rr cc cr cs sr
P X r X c P X c X c
P X r X s P X s X c
p p p p p p
p p p p p p
claim: (P2)cr = (PP)cr =
(P2)ij =
P2 = P(2)
Pm = P(m)
21
mm-Step Transition Probability Matrix-Step Transition Probability Matrix
pcr pcc pcs
prr
pcr
psr
r
c
s
r c s
(2)crp
(2)ijp
(2)cr rr cc cr cs srijp p p p p p p