1 Introduction to Stochastic Models GSLM 54100. 2 Outline continuous-time Markov chain.
-
Upload
gladys-ella-mcdowell -
Category
Documents
-
view
217 -
download
0
Transcript of 1 Introduction to Stochastic Models GSLM 54100. 2 Outline continuous-time Markov chain.
1
Introduction to Stochastic ModelsIntroduction to Stochastic ModelsGSLM 54100GSLM 54100
2
OutlineOutline
continuous-time Markov chain
3
Continuous-Time Markov ChainContinuous-Time Markov Chain continuous-time analog of discrete-time Markov chain objective: to deduce the long-term average cost per unit
time
Discrete-time Markov chain {Xn}
Continuous-time Markov chain {X(t)}
State
Markov property
Stationary transition
Discrete Discrete
P(Xn+1 = j|Xn = i, Xn-1, …, X0) = P(Xn+1 = j|Xn = i) i, j, n
P(X(t+s) = j|X(t) = i, X(r) = x(r), 0 r < t) = P(X(t+s) = j|X(t) = i), i, j, t, s (> 0)
P(Xn+1 = j|Xn = i) = pij , i, j, n
P(X(t+s) = j|X(t) = i) = pij(s) i, j, t, s (> 0)
4
Differences Between Differences Between CTMCCTMC & & DTMCDTMC
duration of a state DTMC: one period
the consecutive duration staying in a state ~ Geo
CTMC: any positive real number the consecutive duration staying in a state ~ exp
cost calculation DTMC: cost per visit = cost per unit time at state
CTMC: cost per visit cost per unit time at state
5
Cost Calculation in Cost Calculation in CTMCCTMC
ci = cost per unit at state i
long-term average cost per unit time
{ ( ) }01lim ,
ti X s i
i
t
c ds
t
{ ( ) }
1, if ( ) , where 1 =
0, . .X s iX s i
o w
ci = cost per visit of state i
long-term average cost per unit time (total number of visits to state in [0, t])
limi
i
t
c i
t
6
Sojourn Times of Sojourn Times of CTMCCTMC
sojourn time: the time that a CTMC stays at a state in a visit
fact:
result of the Markov property
time
excellent
good
fair
bad
1eX
1gX
2eX
2gX
( ) ~ i.i.d. exp( )iX
7
Markov Property of Markov Property of CTMCCTMC
P(X(t+s) = j |X(t) = i, X(r) = x(r) for 0 r < t) = P(X(t+s) = j | X(t) = i) for all i, j and for all t, s > 0 given present (i.e., X(t)), the future (i.e., X(t+s))
and the past (i.e., X(r) = x(r) for 0 r < t) are independent
0 t t+s
X(t) = i X(t+s) = ?
known states, X(r) = x(r), 0 r < t
present future
past
8
Intuitive Justification Intuitive Justification of i.i.d. Exp Sojourn Timesof i.i.d. Exp Sojourn Times
Ti: the sojourn time of state i
just entering state i at t = 0
P(Ti > t) = P(Ti > t|X(0) = i) being a function of i
suppose X(r) = i for 0 r t P(Ti > t+s|X(r) = i, 0 r t) = P(Ti > t+s|Ti > t)
Markov property P(Ti > t+s|X(r) = i, 0 r s) = P(Ti > t|X(0) = i) = P(Ti > t)
P(Ti > t+s|Ti > t) = P(Ti > t) Ti ~ exp(qi)
9
Definition of a Definition of a CTMCCTMC
{X(t)}, a continuous-time, discrete-state stochastic process is a CTMC if
(a). sojourn times at state i, Ti ~ exp(qi), independent of everything else, 0 < qi <
(b). whenever leaving state i, next visiting state j with probability pij > 0, j pij = 1
for the time being, assume pii = 0 for all i, because of the way that
we define sojourn times
10
Procedure to Define a Procedure to Define a CTMCCTMC
defining three kinds of quantities the states (and the state space)
qi for all i (where Ti ~ exp(qi)) and
pij for all i, j (where next visiting j with probability pij upon leaving i )
11
Example 8.4.2Example 8.4.2
one machine, one repairman
working times of machine ~ i.i.d. exp()
repairing times of machine ~ i.i.d. exp()
working
repairing
12
Example 8.4.3Example 8.4.3
two machines, one repairman
working times of all machines ~ i.i.d. exp()
repairing times of all machine ~ i.i.d. exp()
state N(t) = number of working machines
1
0
2
N(t)
t
exp(2)
exp() exp()
exp(+) exp(+)
state 2: two exp() competing
state 1: one exp() and one exp() competing
state 0: one exp() competing
13
Another WayAnother Wayto Define a to Define a CTMCCTMC
last example revealing that in a CTMC, events with exponential times to occur competing to be the first one
defining three kinds of quantities the states (and the state space)
qij for all i, j (where Tij ~ exp(qij) is the duration of the activity that leads to state change from i to j)
qi, where Ti = min(Tij, j i) ~ exp(qi) and qi = j qij
14
Two Ways to Define a Two Ways to Define a CTMCCTMC
{qi, {pij}} { Tij ~ exp(qij), Ti = min(Tij, j i)}
Tij ~ exp(qij)Ti = min(Tij, j
i)
qi = j qij,pij = P(Ti = min(Tij, j i)) = qij/qi
Ti ~ exp(qi)
qi, {pij}
occurrence of events following Poisson of rate qi
random partitioning
by pij
time to change to state j,
Ti1 ~ exp(qi pi1),…
Tij ~ exp(qi pij)…
15
Transition DiagramTransition Diagram
Example 8.4.2: Example 8.4.2: NN((tt) = # of working machine ) = # of working machine at at tt
Example 8.4.3: Example 8.4.3: NN((tt) = # of working machine ) = # of working machine at at tt
0
1
0
12
2
16
Birth and Death ProcessesBirth and Death Processes
a CTMC such that qi,j = 0 for |i-j| 1
birth rate at state i: i
death rate at state i: i
2102
1
1
0… n
2 nn1
3 n+1n
17
Standard Standard MM//MM/1 Queue/1 Queue
Poisson arrivals of rate , exponential service of rate , single server, infinite buffer
210
… n
18
MM//MM/1 Queue with Finite Buffer/1 Queue with Finite Buffer
Poisson arrivals of rate , exponential service of rate , single server, finite buffer of at most N customers in system
210
…
NN1
19
Standard Standard MM//MM//cc Queue Queue
Poisson arrivals of rate , exponential service of rate , c servers, infinite buffer
210
…
3
c+1c
2
c1
(c-1) c
c
c
…
20
MM//MM//cc Queue with Finite Buffer Queue with Finite Buffer
Poisson arrivals of rate , exponential service of rate , c servers, finite buffer of size N, N c
210
… c1
(c-1)2
3
c
c
c+1
c…
cN1
cN
c
21
MM//MM// Queue Queue
Poisson arrivals of rate , exponential service of rate , number of servers, infinite buffer
210
…
2
3
3
4
4
5
22
Finite-Source Queue Finite-Source Queue with Exponential Serverswith Exponential Servers
K machines, i.i.d. working times ~ exp()
R repairmen, i.i.d. repairing times ~ exp()
210
K… R1
(R-1)2 3R
R
R+1
R…
RN1
2
RN
R
(K-1) (K-2) (K-R+2) (K-R+1) (K-R) (K-R-1)
23
Limiting Behavior Limiting Behavior of a Positive Irreducible of a Positive Irreducible DTMCDTMC
an irreducible DTMC {Xn} is positive there exists a unique nonnegative solution to
j: stationary (steady-state) distribution of {Xn}
0
0
1 (normalization eqt)
, for all , (balance eqts)
jj
j i iji
p j
24
Limiting Behavior Limiting Behavior of a Positive Irreducible of a Positive Irreducible DTMCDTMC
j = fraction of time at state j
j = fraction of expected time at state j
average cost cj for each visit at state j
random i.i.d. Cj for each visit at state j
for aperiodic chain:
1
0lim
k
n
Xk
j jn j
E cc
n
1
0lim ( )
k
n
Xk
j jn j
E CE C
n
0lim ( | )n jn
P X j X
25
Limiting Behavior Limiting Behavior of a Positive Irreducible of a Positive Irreducible CTMCCTMC
if {pj} satisfy
cj = cost/time at j
dj = cost/visit at j
for all
1
j j i iji j
ii
p q p q j
p
( )01lim ,
the fraction of time at state
tX s j
jt
dsp
tj
lim ( ( ) | (0) )jt
p P X t j X i
( )01
lim
tj X s j
jj j
t j
c ds
c pt
(# of visits to state in [0, ])
lim j
jj j j
t j
d j t
d q pt
26
Intuitive Derivation Intuitive Derivation of Expressions for Average Costof Expressions for Average Cost
for cj, cost per unit time
T: a large positive number
time at state j pjT
cost induced by state j cjpjT
total cost j cjpjT
cost per unit time = j cjpj
27
Intuitive Derivation Intuitive Derivation of Expressions for Average Costof Expressions for Average Cost
for dj, cost per visit T: a large positive number
time at state j pjT
mean time in state j per visit = 1/qj
approximate number of visits in pjT qjpjT
cost induced by state j cjqjpjT
total cost j cjqjpj
cost per unit time = j cjqjpj
28
ExamplesExamples
Example 8.6.1 Example 8.6.1
stationary distributions of the queueing stationary distributions of the queueing systemssystems
M/M/1
M/M/c
M/M/
M/M/1 with finite buffer
M/M/c with finite buffer
finite-source queue with exponential servers
29
Expected Time Taken Expected Time Taken to Visit a State to Visit a State
XX(0) = 0(0) = 0
00 = = EE(time taken to visit state 2 for the first (time taken to visit state 2 for the first
time|time|XX(0) = 0) (0) = 0)
define define 11 = = EE(time taken to visit state 2 for the (time taken to visit state 2 for the
first time|first time|XX(0) = 0) (0) = 0)
210
2
0 11
1 01
30
Exercise 8.4.8.
The arrivals of potential customers (cars) to a two-mechanic garage follow a Poisson process of rate . The garage has space for two cars; cars that find the garage full go away and never return. At any moment, a car service requires only one mechanic.
31
Exercise 8.4.8.
(a). Suppose that service times are i.i.d. exponential random duration of rate , and that the service and the arrival processes are independent.
(i). Model the problem as a CTMC. You may draw out the transition diagram of the chain after you have defined the state and state space.
(ii). The chain should be positive. Find its stationary distribution.
(iii). Each served car pays a lump sum of $c for its service, and the garage pays each mechanic $r per unit time. Find the long-term net revenue rate ($ per unit time) of the garage.
(iv). In addition to (iii), suppose that the garage pays back a served car $h for each unit time that the car stays in the garage. (Woo! A car can actually earn some money if its service is longer than c/h.) Find the long-term net revenue rate of the garage in this case.
32
Exercise 8.4.8.
Model the following parts as CTMCs. Each part should be modeled independently from others and from part (a). For each part, first define the state and state space and then draw its transition diagram.
(b). There are two. Any service completed by the first mechanic is of exp(1) and that by the second mechanic of exp(2). When a car comes, if only one mechanics is idle, the mechanic picks up the job; if both mechanics are idle, the car is assigned to the mechanic who has taken longer rest since last service completion.
33
Exercise 8.4.8.
(c). There are two types of service, A and B, provided by the garage. For any of the two mechanics, a type A service ~ exp(A) while type B ~ exp(B). Each car needs only one type of service: type A with probability p, 0 < p < 1, independent of everything else, and type B otherwise. The service times are independent from each other and from the arrival process.