Lesson 11: Markov Chains
-
Upload
matthew-leingang -
Category
Technology
-
view
11.506 -
download
0
description
Transcript of Lesson 11: Markov Chains
![Page 1: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/1.jpg)
Lesson 11Markov Chains
Math 20
October 15, 2007
Announcements
I Review Session (ML), 10/16, 7:30–9:30 Hall E
I Problem Set 4 is on the course web site. Due October 17
I Midterm I 10/18, Hall A 7–8:30pm
I OH: Mondays 1–2, Tuesdays 3–4, Wednesdays 1–3 (SC 323)
I Old exams and solutions on website
![Page 2: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/2.jpg)
The Markov Dance
Divide the class into three groups A, B, and C .
Upon my signal:
I 1/3 of group A goes to group B, and 1/3 of group A goes togroup C .
I 1/4 of group B goes to group A, and 1/4 of group A goes togroup C .
I 1/2 of group C goes to group B.
![Page 3: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/3.jpg)
The Markov Dance
Divide the class into three groups A, B, and C . Upon my signal:
I 1/3 of group A goes to group B, and 1/3 of group A goes togroup C .
I 1/4 of group B goes to group A, and 1/4 of group A goes togroup C .
I 1/2 of group C goes to group B.
![Page 4: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/4.jpg)
The Markov Dance
Divide the class into three groups A, B, and C . Upon my signal:
I 1/3 of group A goes to group B, and 1/3 of group A goes togroup C .
I 1/4 of group B goes to group A, and 1/4 of group A goes togroup C .
I 1/2 of group C goes to group B.
![Page 5: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/5.jpg)
The Markov Dance
Divide the class into three groups A, B, and C . Upon my signal:
I 1/3 of group A goes to group B, and 1/3 of group A goes togroup C .
I 1/4 of group B goes to group A, and 1/4 of group A goes togroup C .
I 1/2 of group C goes to group B.
![Page 6: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/6.jpg)
Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007
Page2of9
![Page 7: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/7.jpg)
Another Example
Suppose on any given class day you wake up and decide whether tocome to class. If you went to class the time before, you’re 70%likely to go today, and if you skipped the last class, you’re 80%likely to go today.
Some questions you might ask are:
I If I go to class on Monday, how likely am I to go to class onFriday?
I Assuming the class is infinitely long (the horror!),approximately what portion of class will I attend?
![Page 8: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/8.jpg)
Another Example
Suppose on any given class day you wake up and decide whether tocome to class. If you went to class the time before, you’re 70%likely to go today, and if you skipped the last class, you’re 80%likely to go today. Some questions you might ask are:
I If I go to class on Monday, how likely am I to go to class onFriday?
I Assuming the class is infinitely long (the horror!),approximately what portion of class will I attend?
![Page 9: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/9.jpg)
Another Example
Suppose on any given class day you wake up and decide whether tocome to class. If you went to class the time before, you’re 70%likely to go today, and if you skipped the last class, you’re 80%likely to go today. Some questions you might ask are:
I If I go to class on Monday, how likely am I to go to class onFriday?
I Assuming the class is infinitely long (the horror!),approximately what portion of class will I attend?
![Page 10: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/10.jpg)
Many times we are interested in the transition of somethingbetween certain “states” over discrete time steps. Examples are
I movement of people between regions
I states of the weather
I movement between positions on a Monopoly board
I your score in blackjack
DefinitionA Markov chain or Markov process is a process in which theprobability of the system being in a particular state at a givenobservation period depends only on its state at the immediatelypreceding observation period.
![Page 11: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/11.jpg)
Many times we are interested in the transition of somethingbetween certain “states” over discrete time steps. Examples are
I movement of people between regions
I states of the weather
I movement between positions on a Monopoly board
I your score in blackjack
DefinitionA Markov chain or Markov process is a process in which theprobability of the system being in a particular state at a givenobservation period depends only on its state at the immediatelypreceding observation period.
![Page 12: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/12.jpg)
Common questions about a Markov chain are:
I What is the probability of transitions from state to state overmultiple observations?
I Are there any “equilibria” in the process?
I Is there a long-term stability to the process?
![Page 13: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/13.jpg)
DefinitionSuppose the system has n possible states. For each i and j , let tijbe the probability of switching from state j to state i . The matrixT whose ijth entry is tij is called the transition matrix.
Example
The transition matrix for the skipping class example is
T =
(0.7 0.80.3 0.2
)
![Page 14: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/14.jpg)
DefinitionSuppose the system has n possible states. For each i and j , let tijbe the probability of switching from state j to state i . The matrixT whose ijth entry is tij is called the transition matrix.
Example
The transition matrix for the skipping class example is
T =
(0.7 0.80.3 0.2
)
![Page 15: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/15.jpg)
Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007
Page3of9
![Page 16: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/16.jpg)
Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007
Page4of9
![Page 17: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/17.jpg)
The big idea about the transition matrix reflects an important factabout probabilities:
I All entries are nonnegative.
I The columns add up to one.
Such a matrix is called a stochastic matrix.
![Page 18: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/18.jpg)
DefinitionThe state vector of a Markov process with n-states at time step kis the vector
x(k) =
p
(k)1
p(k)2...
p(k)n
where p
(k)j is the probability that the system is in state j at time
step k .
Example
Suppose we start out with 20 students in group A and 10 studentsin groups B and C . Then the initial state vector is
x(0) =
0.50.250.25
.
![Page 19: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/19.jpg)
Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007
Page5of9
![Page 20: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/20.jpg)
DefinitionThe state vector of a Markov process with n-states at time step kis the vector
x(k) =
p
(k)1
p(k)2...
p(k)n
where p
(k)j is the probability that the system is in state j at time
step k .
Example
Suppose we start out with 20 students in group A and 10 studentsin groups B and C . Then the initial state vector is
x(0) =
0.50.250.25
.
![Page 21: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/21.jpg)
DefinitionThe state vector of a Markov process with n-states at time step kis the vector
x(k) =
p
(k)1
p(k)2...
p(k)n
where p
(k)j is the probability that the system is in state j at time
step k .
Example
Suppose we start out with 20 students in group A and 10 studentsin groups B and C . Then the initial state vector is
x(0) =
0.50.250.25
.
![Page 22: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/22.jpg)
Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007
Page6of9
![Page 23: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/23.jpg)
Example
Suppose after three weeks of class I am equally likely to come to
class or skip. Then my state vector would be x(10) =
(0.50.5
)The big idea about state vectors reflects an important fact aboutprobabilities:
I All entries are nonnegative.
I The entries add up to one.
Such a vector is called a probability vector.
![Page 24: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/24.jpg)
Example
Suppose after three weeks of class I am equally likely to come to
class or skip. Then my state vector would be x(10) =
(0.50.5
)
The big idea about state vectors reflects an important fact aboutprobabilities:
I All entries are nonnegative.
I The entries add up to one.
Such a vector is called a probability vector.
![Page 25: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/25.jpg)
Example
Suppose after three weeks of class I am equally likely to come to
class or skip. Then my state vector would be x(10) =
(0.50.5
)The big idea about state vectors reflects an important fact aboutprobabilities:
I All entries are nonnegative.
I The entries add up to one.
Such a vector is called a probability vector.
![Page 26: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/26.jpg)
LemmaLet T be an n × n stochastic matrix and x an n × 1 probabilityvector. Then Tx is a probability vector.
Proof.We need to show that the entries of Tx add up to one. We have
n∑i=1
(Tx)i =n∑
i=1
n∑j=1
tijxj
=n∑
j=1
(n∑
i=1
tij
)xj
=n∑
j=1
1 · xj = 1
![Page 27: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/27.jpg)
LemmaLet T be an n × n stochastic matrix and x an n × 1 probabilityvector. Then Tx is a probability vector.
Proof.We need to show that the entries of Tx add up to one. We have
n∑i=1
(Tx)i =n∑
i=1
n∑j=1
tijxj
=n∑
j=1
(n∑
i=1
tij
)xj
=n∑
j=1
1 · xj = 1
![Page 28: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/28.jpg)
TheoremIf T is the transition matrix of a Markov process, then the statevector x(k+1) at the (k + 1)th observation period can bedetermined from the state vector x(k) at the kth observationperiod, as
x(k+1) = Tx(k)
This comes from an important idea in conditional probability:
P(state i at t = k + 1)
=n∑
j=1
P(move from state j to state i)P(state j at t = k)
That is, for each i ,
p(k+1)i =
n∑j=1
tijp(k)j
![Page 29: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/29.jpg)
TheoremIf T is the transition matrix of a Markov process, then the statevector x(k+1) at the (k + 1)th observation period can bedetermined from the state vector x(k) at the kth observationperiod, as
x(k+1) = Tx(k)
This comes from an important idea in conditional probability:
P(state i at t = k + 1)
=n∑
j=1
P(move from state j to state i)P(state j at t = k)
That is, for each i ,
p(k+1)i =
n∑j=1
tijp(k)j
![Page 30: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/30.jpg)
Illustration
Example
How does the probability of going to class on Wednesday dependon the probabilities of going to class on Monday?
go skip
go skip go skip
p(k)1
t11 t21
p(k)2
t12 t22
Monday
Wednesday
p(k+1)1 = t11p
(k)1 + t12p
(k)2
p(k+1)2 = t21p
(k)1 + t22p
(k)2
![Page 31: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/31.jpg)
Example
If I go to class on Monday, what’s the probability I’ll go to class onFriday?
Solution
We have x(0) =
(10
). We want to know x(2). We have
x(2) = Tx(1) = T (Tx(0)) = T 2 = Tx(0)
=
(0.7 0.80.3 0.2
)2(10
)=
(0.7 0.80.3 0.2
)(0.70.3
)=
(0.730.27
)
![Page 32: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/32.jpg)
Example
If I go to class on Monday, what’s the probability I’ll go to class onFriday?
Solution
We have x(0) =
(10
). We want to know x(2). We have
x(2) = Tx(1) = T (Tx(0)) = T 2 = Tx(0)
=
(0.7 0.80.3 0.2
)2(10
)=
(0.7 0.80.3 0.2
)(0.70.3
)=
(0.730.27
)
![Page 33: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/33.jpg)
Let’s look at successive powers of the probability matrix. Do theyconverge? To what?
![Page 34: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/34.jpg)
Let’s look at successive powers of the transition matrix in theMarkov Dance.
T =
0.333333 0.25 0.0.333333 0.5 0.50.333333 0.25 0.5
T 2 =
0.194444 0.208333 0.1250.444444 0.458333 0.50.361111 0.333333 0.375
T 3 =
0.175926 0.184028 0.1666670.467593 0.465278 0.4791670.356481 0.350694 0.354167
![Page 35: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/35.jpg)
Let’s look at successive powers of the transition matrix in theMarkov Dance.
T =
0.333333 0.25 0.0.333333 0.5 0.50.333333 0.25 0.5
T 2 =
0.194444 0.208333 0.1250.444444 0.458333 0.50.361111 0.333333 0.375
T 3 =
0.175926 0.184028 0.1666670.467593 0.465278 0.4791670.356481 0.350694 0.354167
![Page 36: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/36.jpg)
Let’s look at successive powers of the transition matrix in theMarkov Dance.
T =
0.333333 0.25 0.0.333333 0.5 0.50.333333 0.25 0.5
T 2 =
0.194444 0.208333 0.1250.444444 0.458333 0.50.361111 0.333333 0.375
T 3 =
0.175926 0.184028 0.1666670.467593 0.465278 0.4791670.356481 0.350694 0.354167
![Page 37: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/37.jpg)
T 4 =
0.17554 0.177662 0.1753470.470679 0.469329 0.4722220.353781 0.353009 0.352431
T 5 =
0.176183 0.176553 0.1765050.470743 0.47039 0.4707750.353074 0.353057 0.35272
T 6 =
0.176414 0.176448 0.1765290.470636 0.470575 0.4705830.35295 0.352977 0.352889
Do they converge? To what?
![Page 38: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/38.jpg)
T 4 =
0.17554 0.177662 0.1753470.470679 0.469329 0.4722220.353781 0.353009 0.352431
T 5 =
0.176183 0.176553 0.1765050.470743 0.47039 0.4707750.353074 0.353057 0.35272
T 6 =
0.176414 0.176448 0.1765290.470636 0.470575 0.4705830.35295 0.352977 0.352889
Do they converge? To what?
![Page 39: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/39.jpg)
T 4 =
0.17554 0.177662 0.1753470.470679 0.469329 0.4722220.353781 0.353009 0.352431
T 5 =
0.176183 0.176553 0.1765050.470743 0.47039 0.4707750.353074 0.353057 0.35272
T 6 =
0.176414 0.176448 0.1765290.470636 0.470575 0.4705830.35295 0.352977 0.352889
Do they converge? To what?
![Page 40: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/40.jpg)
A transition matrix (or corresponding Markov process) is calledregular if some power of the matrix has all nonzero entries. Or,there is a positive probability of eventually moving from every stateto every state.
![Page 41: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/41.jpg)
Theorem 2.5If T is the transition matrix of a regular Markov process, then
(a) As n→∞, T n approaches a matrix
A =
u1 u1 . . . u1
u2 u2 . . . u2
. . . . . . . . . . . .un un . . . un
,
all of whose columns are identical.
(b) Every column u is a a probability vector all of whosecomponents are positive.
![Page 42: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/42.jpg)
Theorem 2.6If T is a regular∗ transition matrix and A and u are as above, then
(a) For any probability vector x, Tnx→ u as n→∞, so that u isa steady-state vector.
(b) The steady-state vector u is the unique probability vectorsatisfying the matrix equation Tu = u.
![Page 43: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/43.jpg)
Finding the steady-state vector
We know the steady-state vector is unique. So we use the equationit satisfies to find it: Tu = u.
This is a matrix equation if you put it in the form
(T− I)u = 0
![Page 44: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/44.jpg)
Finding the steady-state vector
We know the steady-state vector is unique. So we use the equationit satisfies to find it: Tu = u.This is a matrix equation if you put it in the form
(T− I)u = 0
![Page 45: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/45.jpg)
Math 20 - October 15, 2007.GWBMonday, Oct 15, 2007
Page9of9
![Page 46: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/46.jpg)
Example (Skipping class)
If the transition matrix is T =
(0.7 0.80.3 0.2
), what is the
steady-state vector?
SolutionWe can combine the equations (T − I )u = 0, u1 + u2 = 1 into asingle linear system with augmented matrix −3/10 8/10 0
3/10 −8/10 01 1 1
1 0 8/11
0 1 3/11
0 0 0
So the steady-state vector is
(8/11
3/11
). You’ll go to class about 72%
of the time.
![Page 47: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/47.jpg)
Example (Skipping class)
If the transition matrix is T =
(0.7 0.80.3 0.2
), what is the
steady-state vector?
SolutionWe can combine the equations (T − I )u = 0, u1 + u2 = 1 into asingle linear system with augmented matrix −3/10 8/10 0
3/10 −8/10 01 1 1
1 0 8/11
0 1 3/11
0 0 0
So the steady-state vector is
(8/11
3/11
). You’ll go to class about 72%
of the time.
![Page 48: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/48.jpg)
Example (Skipping class)
If the transition matrix is T =
(0.7 0.80.3 0.2
), what is the
steady-state vector?
SolutionWe can combine the equations (T − I )u = 0, u1 + u2 = 1 into asingle linear system with augmented matrix −3/10 8/10 0
3/10 −8/10 01 1 1
1 0 8/11
0 1 3/11
0 0 0
So the steady-state vector is
(8/11
3/11
). You’ll go to class about 72%
of the time.
![Page 49: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/49.jpg)
Example (The Markov Dance)
If the transition matrix is T =
1/3 1/4 01/3 1/2 1/2
1/3 1/4 1/2
, what is the
steady-state vector?
SolutionWe have
−2/3 1/4 0 01/3 −1/2 1/2 01/3 1/4 −1/2 0
1 1 1 1
1 0 0 3/17
0 1 0 8/17
0 0 1 6/17
0 0 0 0
![Page 50: Lesson 11: Markov Chains](https://reader034.fdocuments.us/reader034/viewer/2022051109/54798b905906b507358b458a/html5/thumbnails/50.jpg)
Example (The Markov Dance)
If the transition matrix is T =
1/3 1/4 01/3 1/2 1/2
1/3 1/4 1/2
, what is the
steady-state vector?
SolutionWe have
−2/3 1/4 0 01/3 −1/2 1/2 01/3 1/4 −1/2 0
1 1 1 1
1 0 0 3/17
0 1 0 8/17
0 0 1 6/17
0 0 0 0