Discrete Time Markov Chains EE384X Review 2 Winter 2006.
-
Upload
beatrice-bond -
Category
Documents
-
view
220 -
download
3
Transcript of Discrete Time Markov Chains EE384X Review 2 Winter 2006.
![Page 1: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/1.jpg)
Discrete Time Markov Chains
EE384X Review 2
Winter 2006
![Page 2: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/2.jpg)
Outline
• Some examples
• Definitions
• Stationary Distributions
References (on reserve in library):
1. Hoel, Port, and Stone: Introduction to Stochastic Processes
2. Wolff: Stochastic Modeling and the Theory of Queues
![Page 3: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/3.jpg)
Simple DTMCs
• “States” can be labeled (0,)1,2,3,…
• At every time slot a “jump” decision is made randomly based on current state
•
10
p
q
1-q1-p
10
2
a
d fc b
e
(Sometimes the arrow pointing back to the same state is omitted)
![Page 4: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/4.jpg)
1-D Random Walk
• Time is slotted
• The walker flips a coin every time slot to decide which way to go
•
X(t)
p1-p
![Page 5: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/5.jpg)
Single Server Queue
• Consider a queue at a supermarket
• In every time slot:– A customer arrives with probability p– The HoL customer leaves with probability q
Bernoulli(p)Geom(q)
![Page 6: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/6.jpg)
Birth-Death Chain
• Can be modeled by a Birth-Death Chain (aka. Geom/Geom/1 queue)
• Want to know: – Queue size distribution – Average waiting time, etc.
0 1 2 3
![Page 7: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/7.jpg)
Markov Property
• “Future” is independent of “Past” given “Present”
• In other words: Memoryless
• We’ve seen memoryless distributions: Exponential and Geometric
• Useful for modeling and analyzing real systems
![Page 8: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/8.jpg)
Discrete Time Markov Chains
• A sequence of random variables {Xn} is called a Markov chain if it has the Markov property:
• States are usually labeled {(0,)1,2,…}
• State space can be finite or infinite
![Page 9: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/9.jpg)
Transition Probability
• Probability to jump from state i to state j
• Assume stationary: independent of time
• Transition probability matrix:
P = (pij)
• Two state MC:
![Page 10: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/10.jpg)
Stationary Distribution
Define
Then k+1 = k P ( is a row vector)
Stationary Distribution:
if the limit exists.
If exists, we can solve it by
![Page 11: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/11.jpg)
Balance Equations
• These are called balance equations– Transitions in and out of state i are balanced
![Page 12: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/12.jpg)
In General
• If we partition all the states into two sets, then transitions between the two sets must be “balanced”.– Equivalent to a bi-section in the state transition
graph– This can be easily derived from the Balance
Equations
![Page 13: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/13.jpg)
Conditions for to Exist (I)
• Definitions:– State j is reachable by state i if
– State i and j commute if they are reachable by each other
– The Markov chain is irreducible if all states commute
![Page 14: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/14.jpg)
Conditions for to Exist (I) (cont’d)
• Condition: The Markov chain is irreducible
• Counter-examples:21 43
32p=1
1
![Page 15: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/15.jpg)
Conditions for to Exist (II)
• The Markov chain is aperiodic:
• Counter-example:
10
2
1
0 01 1
0
![Page 16: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/16.jpg)
Conditions for to Exist (III)
• The Markov chain is positive recurrent:– State i is recurrent if
– Otherwise transient– If recurrent
• State i is positive recurrent if E(Ti)<1, where Ti is time between visits to state i
• Otherwise null recurrent
![Page 17: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/17.jpg)
Solving for
10
p
q
1-q1-p
![Page 18: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/18.jpg)
Birth-Death Chain
• Arrival w.p. p ; departure w.p. q
• Let u = p(1-q), d = q(1-p), = u/d
• Balance equations:
u u u u
d d d d
0 1 2 31-u
1-u-d 1-u-d 1-u-d
![Page 19: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/19.jpg)
Birth-Death Chain (cont’d)
• Continue like this, we can derive:
(i-1) u = (i) d
• Equivalently, we can draw a bi-section between state i and state i-1
• Therefore, we have
![Page 20: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/20.jpg)
Birth-Death Chain (cont’d)
![Page 21: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/21.jpg)
Any Problems?
• What if is greater than 1?– Then the stationary distribution does not exist
• Which condition does it violate?
![Page 22: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/22.jpg)
2£2 Switch w/ HoL Blocking
• Packets arrive as Bernoulli iid uniform • Packets queued at inputs• Only one packet can leave an output every time slot
1
1
1
2
2
1
1
2
![Page 23: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/23.jpg)
2£2 Switch (cont’d)
• If both HoL packets are destined to the same output– Only one of them is served (chosen randomly)– The other output is idle, as packets are blocked– This is called head of line blocking– HoL blocking reduces throughput
• Want to know: throughput of this switch
![Page 24: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/24.jpg)
2£2 Switch - DTMC
• States are the number of HoL packets destined to output 1 and output 2
• But states (0,2) and (2,0) are the same– Can “collapse” them together
1,1 2,00,2 0.25 0.25
0.5 0.5
0.5
0.5 0.5
![Page 25: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/25.jpg)
2£2 Switch – DTMC (cont’d)
• Now P{(0,2)} = P{(1,1)} = 0.5
• Switch throughput = 0.5£1+0.5£2 = 1.5
• Per output throughput = 1.5/2 = 0.75
1,10,2
0.5
0.5
0.5 0.5
![Page 26: Discrete Time Markov Chains EE384X Review 2 Winter 2006.](https://reader035.fdocuments.us/reader035/viewer/2022062407/56649cf85503460f949c9822/html5/thumbnails/26.jpg)
Another Method to Find
• Sometimes the Markov chain is not easy to solve analytically
• Can run the Markov chain for a long time, then
{fraction of time spent in state i} ! (i)