18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175...
Transcript of 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175...
![Page 1: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/1.jpg)
18.175: Lecture 21
More Markov chains
Scott Sheffield
MIT
18.175 Lecture 21
![Page 2: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/2.jpg)
Outline
Recollections
General setup and basic properties
Recurrence and transience
18.175 Lecture 21
![Page 3: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/3.jpg)
Outline
Recollections
General setup and basic properties
Recurrence and transience
18.175 Lecture 21
![Page 4: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/4.jpg)
Markov chains
I Consider a sequence of random variables X0,X1,X2, . . . eachtaking values in the same state space, which for now we taketo be a finite set that we label by 0, 1, . . . ,M.
I Interpret Xn as state of the system at time n.
I Sequence is called a Markov chain if we have a fixedcollection of numbers Pij (one for each pairi , j ∈ 0, 1, . . . ,M) such that whenever the system is in statei , there is probability Pij that system will next be in state j .
I Precisely,PXn+1 = j |Xn = i ,Xn−1 = in−1, . . . ,X1 = i1,X0 = i0 = Pij .
I Kind of an “almost memoryless” property. Probabilitydistribution for next state depends only on the current state(and not on the rest of the state history).
18.175 Lecture 21
![Page 5: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/5.jpg)
Markov chains
I Consider a sequence of random variables X0,X1,X2, . . . eachtaking values in the same state space, which for now we taketo be a finite set that we label by 0, 1, . . . ,M.
I Interpret Xn as state of the system at time n.
I Sequence is called a Markov chain if we have a fixedcollection of numbers Pij (one for each pairi , j ∈ 0, 1, . . . ,M) such that whenever the system is in statei , there is probability Pij that system will next be in state j .
I Precisely,PXn+1 = j |Xn = i ,Xn−1 = in−1, . . . ,X1 = i1,X0 = i0 = Pij .
I Kind of an “almost memoryless” property. Probabilitydistribution for next state depends only on the current state(and not on the rest of the state history).
18.175 Lecture 21
![Page 6: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/6.jpg)
Markov chains
I Consider a sequence of random variables X0,X1,X2, . . . eachtaking values in the same state space, which for now we taketo be a finite set that we label by 0, 1, . . . ,M.
I Interpret Xn as state of the system at time n.
I Sequence is called a Markov chain if we have a fixedcollection of numbers Pij (one for each pairi , j ∈ 0, 1, . . . ,M) such that whenever the system is in statei , there is probability Pij that system will next be in state j .
I Precisely,PXn+1 = j |Xn = i ,Xn−1 = in−1, . . . ,X1 = i1,X0 = i0 = Pij .
I Kind of an “almost memoryless” property. Probabilitydistribution for next state depends only on the current state(and not on the rest of the state history).
18.175 Lecture 21
![Page 7: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/7.jpg)
Markov chains
I Consider a sequence of random variables X0,X1,X2, . . . eachtaking values in the same state space, which for now we taketo be a finite set that we label by 0, 1, . . . ,M.
I Interpret Xn as state of the system at time n.
I Sequence is called a Markov chain if we have a fixedcollection of numbers Pij (one for each pairi , j ∈ 0, 1, . . . ,M) such that whenever the system is in statei , there is probability Pij that system will next be in state j .
I Precisely,PXn+1 = j |Xn = i ,Xn−1 = in−1, . . . ,X1 = i1,X0 = i0 = Pij .
I Kind of an “almost memoryless” property. Probabilitydistribution for next state depends only on the current state(and not on the rest of the state history).
18.175 Lecture 21
![Page 8: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/8.jpg)
Markov chains
I Consider a sequence of random variables X0,X1,X2, . . . eachtaking values in the same state space, which for now we taketo be a finite set that we label by 0, 1, . . . ,M.
I Interpret Xn as state of the system at time n.
I Sequence is called a Markov chain if we have a fixedcollection of numbers Pij (one for each pairi , j ∈ 0, 1, . . . ,M) such that whenever the system is in statei , there is probability Pij that system will next be in state j .
I Precisely,PXn+1 = j |Xn = i ,Xn−1 = in−1, . . . ,X1 = i1,X0 = i0 = Pij .
I Kind of an “almost memoryless” property. Probabilitydistribution for next state depends only on the current state(and not on the rest of the state history).
18.175 Lecture 21
![Page 9: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/9.jpg)
Matrix representation
I To describe a Markov chain, we need to define Pij for anyi , j ∈ 0, 1, . . . ,M.
I It is convenient to represent the collection of transitionprobabilities Pij as a matrix:
A =
P00 P01 . . . P0M
P10 P11 . . . P1M
···
PM0 PM1 . . . PMM
I For this to make sense, we require Pij ≥ 0 for all i , j and∑M
j=0 Pij = 1 for each i . That is, the rows sum to one.
18.175 Lecture 21
![Page 10: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/10.jpg)
Matrix representation
I To describe a Markov chain, we need to define Pij for anyi , j ∈ 0, 1, . . . ,M.
I It is convenient to represent the collection of transitionprobabilities Pij as a matrix:
A =
P00 P01 . . . P0M
P10 P11 . . . P1M
···
PM0 PM1 . . . PMM
I For this to make sense, we require Pij ≥ 0 for all i , j and∑Mj=0 Pij = 1 for each i . That is, the rows sum to one.
18.175 Lecture 21
![Page 11: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/11.jpg)
Matrix representation
I To describe a Markov chain, we need to define Pij for anyi , j ∈ 0, 1, . . . ,M.
I It is convenient to represent the collection of transitionprobabilities Pij as a matrix:
A =
P00 P01 . . . P0M
P10 P11 . . . P1M
···
PM0 PM1 . . . PMM
I For this to make sense, we require Pij ≥ 0 for all i , j and∑M
j=0 Pij = 1 for each i . That is, the rows sum to one.
18.175 Lecture 21
![Page 12: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/12.jpg)
Powers of transition matrix
I We write P(n)ij for the probability to go from state i to state j
over n steps.
I From the matrix point of view
P(n)00 P
(n)01 . . . P
(n)0M
P(n)10 P
(n)11 . . . P
(n)1M
···
P(n)M0 P
(n)M1 . . . P
(n)MM
=
P00 P01 . . . P0M
P10 P11 . . . P1M
···
PM0 PM1 . . . PMM
n
I If A is the one-step transition matrix, then An is the n-steptransition matrix.
18.175 Lecture 21
![Page 13: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/13.jpg)
Powers of transition matrix
I We write P(n)ij for the probability to go from state i to state j
over n steps.
I From the matrix point of view
P(n)00 P
(n)01 . . . P
(n)0M
P(n)10 P
(n)11 . . . P
(n)1M
···
P(n)M0 P
(n)M1 . . . P
(n)MM
=
P00 P01 . . . P0M
P10 P11 . . . P1M
···
PM0 PM1 . . . PMM
n
I If A is the one-step transition matrix, then An is the n-steptransition matrix.
18.175 Lecture 21
![Page 14: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/14.jpg)
Powers of transition matrix
I We write P(n)ij for the probability to go from state i to state j
over n steps.
I From the matrix point of view
P(n)00 P
(n)01 . . . P
(n)0M
P(n)10 P
(n)11 . . . P
(n)1M
···
P(n)M0 P
(n)M1 . . . P
(n)MM
=
P00 P01 . . . P0M
P10 P11 . . . P1M
···
PM0 PM1 . . . PMM
n
I If A is the one-step transition matrix, then An is the n-steptransition matrix.
18.175 Lecture 21
![Page 15: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/15.jpg)
Ergodic Markov chains
I Say Markov chain is ergodic if some power of the transitionmatrix has all non-zero entries.
I Turns out that if chain has this property, then
πj := limn→∞ P(n)ij exists and the πj are the unique
non-negative solutions of πj =∑M
k=0 πkPkj that sum to one.
I This means that the row vector
π =(π0 π1 . . . πM
)is a left eigenvector of A with eigenvalue 1, i.e., πA = π.
I We call π the stationary distribution of the Markov chain.
I One can solve the system of linear equationsπj =
∑Mk=0 πkPkj to compute the values πj . Equivalent to
considering A fixed and solving πA = π. Or solving(A− I )π = 0. This determines π up to a multiplicativeconstant, and fact that
∑πj = 1 determines the constant.
18.175 Lecture 21
![Page 16: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/16.jpg)
Ergodic Markov chains
I Say Markov chain is ergodic if some power of the transitionmatrix has all non-zero entries.
I Turns out that if chain has this property, then
πj := limn→∞ P(n)ij exists and the πj are the unique
non-negative solutions of πj =∑M
k=0 πkPkj that sum to one.
I This means that the row vector
π =(π0 π1 . . . πM
)is a left eigenvector of A with eigenvalue 1, i.e., πA = π.
I We call π the stationary distribution of the Markov chain.
I One can solve the system of linear equationsπj =
∑Mk=0 πkPkj to compute the values πj . Equivalent to
considering A fixed and solving πA = π. Or solving(A− I )π = 0. This determines π up to a multiplicativeconstant, and fact that
∑πj = 1 determines the constant.
18.175 Lecture 21
![Page 17: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/17.jpg)
Ergodic Markov chains
I Say Markov chain is ergodic if some power of the transitionmatrix has all non-zero entries.
I Turns out that if chain has this property, then
πj := limn→∞ P(n)ij exists and the πj are the unique
non-negative solutions of πj =∑M
k=0 πkPkj that sum to one.
I This means that the row vector
π =(π0 π1 . . . πM
)is a left eigenvector of A with eigenvalue 1, i.e., πA = π.
I We call π the stationary distribution of the Markov chain.
I One can solve the system of linear equationsπj =
∑Mk=0 πkPkj to compute the values πj . Equivalent to
considering A fixed and solving πA = π. Or solving(A− I )π = 0. This determines π up to a multiplicativeconstant, and fact that
∑πj = 1 determines the constant.
18.175 Lecture 21
![Page 18: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/18.jpg)
Ergodic Markov chains
I Say Markov chain is ergodic if some power of the transitionmatrix has all non-zero entries.
I Turns out that if chain has this property, then
πj := limn→∞ P(n)ij exists and the πj are the unique
non-negative solutions of πj =∑M
k=0 πkPkj that sum to one.
I This means that the row vector
π =(π0 π1 . . . πM
)is a left eigenvector of A with eigenvalue 1, i.e., πA = π.
I We call π the stationary distribution of the Markov chain.
I One can solve the system of linear equationsπj =
∑Mk=0 πkPkj to compute the values πj . Equivalent to
considering A fixed and solving πA = π. Or solving(A− I )π = 0. This determines π up to a multiplicativeconstant, and fact that
∑πj = 1 determines the constant.
18.175 Lecture 21
![Page 19: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/19.jpg)
Ergodic Markov chains
I Say Markov chain is ergodic if some power of the transitionmatrix has all non-zero entries.
I Turns out that if chain has this property, then
πj := limn→∞ P(n)ij exists and the πj are the unique
non-negative solutions of πj =∑M
k=0 πkPkj that sum to one.
I This means that the row vector
π =(π0 π1 . . . πM
)is a left eigenvector of A with eigenvalue 1, i.e., πA = π.
I We call π the stationary distribution of the Markov chain.
I One can solve the system of linear equationsπj =
∑Mk=0 πkPkj to compute the values πj . Equivalent to
considering A fixed and solving πA = π. Or solving(A− I )π = 0. This determines π up to a multiplicativeconstant, and fact that
∑πj = 1 determines the constant.
18.175 Lecture 21
![Page 20: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/20.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 21: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/21.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 22: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/22.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 23: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/23.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 24: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/24.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 25: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/25.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 26: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/26.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 27: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/27.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 28: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/28.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 29: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/29.jpg)
Examples
I Random walks on Rd .
I Branching processes: p(i , j) = P(∑i
m=1 ξm = j)
where ξi arei.i.d. non-negative integer-valued random variables.
I Renewal chain (deterministic unit decreases, random jumpwhen zero hit).
I Card shuffling.
I Ehrenfest chain (n balls in two chambers, randomly pick ballto swap).
I Birth and death chains (changes by ±1). Stationaritydistribution?
I M/G/1 queues.
I Random walk on a graph. Stationary distribution?
I Random walk on directed graph (e.g., single directed chain).
I Snakes and ladders.
18.175 Lecture 21
![Page 30: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/30.jpg)
Outline
Recollections
General setup and basic properties
Recurrence and transience
18.175 Lecture 21
![Page 31: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/31.jpg)
Outline
Recollections
General setup and basic properties
Recurrence and transience
18.175 Lecture 21
![Page 32: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/32.jpg)
Markov chains: general definition
I Consider a measurable space (S ,S).
I A function p : S × S → R is a transition probability if
I For each x ∈ S , A→ p(x ,A) is a probability measure on S ,S).I For each A ∈ S , the map x → p(x ,A) is a measurable function.
I Say that Xn is a Markov chain w.r.t. Fn with transitionprobability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I How do we construct an infinite Markov chain? Choose p andinitial distribution µ on (S ,S). For each n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
18.175 Lecture 21
![Page 33: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/33.jpg)
Markov chains: general definition
I Consider a measurable space (S ,S).I A function p : S × S → R is a transition probability if
I For each x ∈ S , A→ p(x ,A) is a probability measure on S ,S).I For each A ∈ S , the map x → p(x ,A) is a measurable function.
I Say that Xn is a Markov chain w.r.t. Fn with transitionprobability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I How do we construct an infinite Markov chain? Choose p andinitial distribution µ on (S ,S). For each n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
18.175 Lecture 21
![Page 34: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/34.jpg)
Markov chains: general definition
I Consider a measurable space (S ,S).I A function p : S × S → R is a transition probability if
I For each x ∈ S , A→ p(x ,A) is a probability measure on S ,S).
I For each A ∈ S , the map x → p(x ,A) is a measurable function.
I Say that Xn is a Markov chain w.r.t. Fn with transitionprobability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I How do we construct an infinite Markov chain? Choose p andinitial distribution µ on (S ,S). For each n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
18.175 Lecture 21
![Page 35: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/35.jpg)
Markov chains: general definition
I Consider a measurable space (S ,S).I A function p : S × S → R is a transition probability if
I For each x ∈ S , A→ p(x ,A) is a probability measure on S ,S).I For each A ∈ S , the map x → p(x ,A) is a measurable function.
I Say that Xn is a Markov chain w.r.t. Fn with transitionprobability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I How do we construct an infinite Markov chain? Choose p andinitial distribution µ on (S ,S). For each n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
18.175 Lecture 21
![Page 36: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/36.jpg)
Markov chains: general definition
I Consider a measurable space (S ,S).I A function p : S × S → R is a transition probability if
I For each x ∈ S , A→ p(x ,A) is a probability measure on S ,S).I For each A ∈ S , the map x → p(x ,A) is a measurable function.
I Say that Xn is a Markov chain w.r.t. Fn with transitionprobability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I How do we construct an infinite Markov chain? Choose p andinitial distribution µ on (S ,S). For each n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
18.175 Lecture 21
![Page 37: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/37.jpg)
Markov chains: general definition
I Consider a measurable space (S ,S).I A function p : S × S → R is a transition probability if
I For each x ∈ S , A→ p(x ,A) is a probability measure on S ,S).I For each A ∈ S , the map x → p(x ,A) is a measurable function.
I Say that Xn is a Markov chain w.r.t. Fn with transitionprobability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I How do we construct an infinite Markov chain? Choose p andinitial distribution µ on (S ,S). For each n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
18.175 Lecture 21
![Page 38: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/38.jpg)
Markov chains
I Definition, again: Say Xn is a Markov chain w.r.t. Fn withtransition probability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I Construction, again: Fix initial distribution µ on (S ,S). Foreach n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
I Notation: Extension produces probability measure Pµ onsequence space (S0,1,...,S0,1,...).
I Theorem: (X0,X1, . . .) chosen from Pµ is Markov chain.
I Theorem: If Xn is any Markov chain with initial distributionµ and transition p, then finite dim. probabilities are as above.
18.175 Lecture 21
![Page 39: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/39.jpg)
Markov chains
I Definition, again: Say Xn is a Markov chain w.r.t. Fn withtransition probability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I Construction, again: Fix initial distribution µ on (S ,S). Foreach n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
I Notation: Extension produces probability measure Pµ onsequence space (S0,1,...,S0,1,...).
I Theorem: (X0,X1, . . .) chosen from Pµ is Markov chain.
I Theorem: If Xn is any Markov chain with initial distributionµ and transition p, then finite dim. probabilities are as above.
18.175 Lecture 21
![Page 40: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/40.jpg)
Markov chains
I Definition, again: Say Xn is a Markov chain w.r.t. Fn withtransition probability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I Construction, again: Fix initial distribution µ on (S ,S). Foreach n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
I Notation: Extension produces probability measure Pµ onsequence space (S0,1,...,S0,1,...).
I Theorem: (X0,X1, . . .) chosen from Pµ is Markov chain.
I Theorem: If Xn is any Markov chain with initial distributionµ and transition p, then finite dim. probabilities are as above.
18.175 Lecture 21
![Page 41: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/41.jpg)
Markov chains
I Definition, again: Say Xn is a Markov chain w.r.t. Fn withtransition probability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I Construction, again: Fix initial distribution µ on (S ,S). Foreach n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
I Notation: Extension produces probability measure Pµ onsequence space (S0,1,...,S0,1,...).
I Theorem: (X0,X1, . . .) chosen from Pµ is Markov chain.
I Theorem: If Xn is any Markov chain with initial distributionµ and transition p, then finite dim. probabilities are as above.
18.175 Lecture 21
![Page 42: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/42.jpg)
Markov chains
I Definition, again: Say Xn is a Markov chain w.r.t. Fn withtransition probability p if P(Xn+1 ∈ B|Fn) = p(Xn,B).
I Construction, again: Fix initial distribution µ on (S ,S). Foreach n <∞ write
P(Xj ∈ Bj , 0 ≤ j ≤ n) =
∫B0
µ(dx0)
∫B1
p(x0, dx1) · · ·
∫Bn
p(xn−1, dxn).
Extend to n =∞ by Kolmogorov’s extension theorem.
I Notation: Extension produces probability measure Pµ onsequence space (S0,1,...,S0,1,...).
I Theorem: (X0,X1, . . .) chosen from Pµ is Markov chain.
I Theorem: If Xn is any Markov chain with initial distributionµ and transition p, then finite dim. probabilities are as above.
18.175 Lecture 21
![Page 43: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/43.jpg)
Markov properties
I Markov property: Take (Ω0,F) =(S0,1,...,S0,1,...
), and
let Pµ be Markov chain measure and θn the shift operator onΩ0 (shifts sequence n units to left, discarding elements shiftedoff the edge). If Y : Ω0 → R is bounded and measurable then
Eµ(Y θn|Fn) = EXnY .
I Strong Markov property: Can replace n with a.s. finitestopping time N and function Y can vary with time. Supposethat for each n, Yn : Ωn → R is measurable and |Yn| ≤ M forall n. Then
Eµ(YN θN |FN) = EXNYN ,
where RHS means ExYn evaluated at x = Xn, n = N.
18.175 Lecture 21
![Page 44: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/44.jpg)
Markov properties
I Markov property: Take (Ω0,F) =(S0,1,...,S0,1,...
), and
let Pµ be Markov chain measure and θn the shift operator onΩ0 (shifts sequence n units to left, discarding elements shiftedoff the edge). If Y : Ω0 → R is bounded and measurable then
Eµ(Y θn|Fn) = EXnY .
I Strong Markov property: Can replace n with a.s. finitestopping time N and function Y can vary with time. Supposethat for each n, Yn : Ωn → R is measurable and |Yn| ≤ M forall n. Then
Eµ(YN θN |FN) = EXNYN ,
where RHS means ExYn evaluated at x = Xn, n = N.
18.175 Lecture 21
![Page 45: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/45.jpg)
Properties
I Property of infinite opportunities: Suppose Xn is Markovchain and
P(∪∞m=n+1Xm ∈ Bm|Xn) ≥ δ > 0
on Xn ∈ An. Then P(Xn ∈ An i .o. − Xn ∈ Bn i .o.) = 0.
I Reflection principle: Symmetric random walks on R. HaveP(supm≥n Sm > a) ≤ 2P(Sn > a).
I Proof idea: Reflection picture.
18.175 Lecture 21
![Page 46: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/46.jpg)
Properties
I Property of infinite opportunities: Suppose Xn is Markovchain and
P(∪∞m=n+1Xm ∈ Bm|Xn) ≥ δ > 0
on Xn ∈ An. Then P(Xn ∈ An i .o. − Xn ∈ Bn i .o.) = 0.
I Reflection principle: Symmetric random walks on R. HaveP(supm≥n Sm > a) ≤ 2P(Sn > a).
I Proof idea: Reflection picture.
18.175 Lecture 21
![Page 47: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/47.jpg)
Properties
I Property of infinite opportunities: Suppose Xn is Markovchain and
P(∪∞m=n+1Xm ∈ Bm|Xn) ≥ δ > 0
on Xn ∈ An. Then P(Xn ∈ An i .o. − Xn ∈ Bn i .o.) = 0.
I Reflection principle: Symmetric random walks on R. HaveP(supm≥n Sm > a) ≤ 2P(Sn > a).
I Proof idea: Reflection picture.
18.175 Lecture 21
![Page 48: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/48.jpg)
Reversibility
I Measure µ called reversible if µ(x)p(x , y) = µ(y)p(y , x) forall x , y .
I Reversibility implies stationarity. Implies that amount of massmoving from x to y is same as amount moving from y to x .Net flow of zero along each edge.
I Markov chain called reversible if admits a reversible probabilitymeasure.
I Are all random walks on (undirected) graphs reversible?
I What about directed graphs?
18.175 Lecture 21
![Page 49: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/49.jpg)
Reversibility
I Measure µ called reversible if µ(x)p(x , y) = µ(y)p(y , x) forall x , y .
I Reversibility implies stationarity. Implies that amount of massmoving from x to y is same as amount moving from y to x .Net flow of zero along each edge.
I Markov chain called reversible if admits a reversible probabilitymeasure.
I Are all random walks on (undirected) graphs reversible?
I What about directed graphs?
18.175 Lecture 21
![Page 50: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/50.jpg)
Reversibility
I Measure µ called reversible if µ(x)p(x , y) = µ(y)p(y , x) forall x , y .
I Reversibility implies stationarity. Implies that amount of massmoving from x to y is same as amount moving from y to x .Net flow of zero along each edge.
I Markov chain called reversible if admits a reversible probabilitymeasure.
I Are all random walks on (undirected) graphs reversible?
I What about directed graphs?
18.175 Lecture 21
![Page 51: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/51.jpg)
Reversibility
I Measure µ called reversible if µ(x)p(x , y) = µ(y)p(y , x) forall x , y .
I Reversibility implies stationarity. Implies that amount of massmoving from x to y is same as amount moving from y to x .Net flow of zero along each edge.
I Markov chain called reversible if admits a reversible probabilitymeasure.
I Are all random walks on (undirected) graphs reversible?
I What about directed graphs?
18.175 Lecture 21
![Page 52: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/52.jpg)
Reversibility
I Measure µ called reversible if µ(x)p(x , y) = µ(y)p(y , x) forall x , y .
I Reversibility implies stationarity. Implies that amount of massmoving from x to y is same as amount moving from y to x .Net flow of zero along each edge.
I Markov chain called reversible if admits a reversible probabilitymeasure.
I Are all random walks on (undirected) graphs reversible?
I What about directed graphs?
18.175 Lecture 21
![Page 53: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/53.jpg)
Cycle theorem
I Kolmogorov’s cycle theorem: Suppose p is irreducible.Then exists reversible measure if and only if
I p(x , y) > 0 implies p(y , x) > 0I for any loop x0, x1, . . . xn with
∏ni=1 p(xi , xi−1) > 0, we have
n∏i=1
p(xi−1, xi )
p(xi , xi−1)= 1.
I Useful idea to have in mind when constructing Markov chainswith given reversible distribution, as needed in Monte CarloMarkov Chains (MCMC) applications.
18.175 Lecture 21
![Page 54: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/54.jpg)
Cycle theorem
I Kolmogorov’s cycle theorem: Suppose p is irreducible.Then exists reversible measure if and only if
I p(x , y) > 0 implies p(y , x) > 0
I for any loop x0, x1, . . . xn with∏n
i=1 p(xi , xi−1) > 0, we have
n∏i=1
p(xi−1, xi )
p(xi , xi−1)= 1.
I Useful idea to have in mind when constructing Markov chainswith given reversible distribution, as needed in Monte CarloMarkov Chains (MCMC) applications.
18.175 Lecture 21
![Page 55: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/55.jpg)
Cycle theorem
I Kolmogorov’s cycle theorem: Suppose p is irreducible.Then exists reversible measure if and only if
I p(x , y) > 0 implies p(y , x) > 0I for any loop x0, x1, . . . xn with
∏ni=1 p(xi , xi−1) > 0, we have
n∏i=1
p(xi−1, xi )
p(xi , xi−1)= 1.
I Useful idea to have in mind when constructing Markov chainswith given reversible distribution, as needed in Monte CarloMarkov Chains (MCMC) applications.
18.175 Lecture 21
![Page 56: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/56.jpg)
Cycle theorem
I Kolmogorov’s cycle theorem: Suppose p is irreducible.Then exists reversible measure if and only if
I p(x , y) > 0 implies p(y , x) > 0I for any loop x0, x1, . . . xn with
∏ni=1 p(xi , xi−1) > 0, we have
n∏i=1
p(xi−1, xi )
p(xi , xi−1)= 1.
I Useful idea to have in mind when constructing Markov chainswith given reversible distribution, as needed in Monte CarloMarkov Chains (MCMC) applications.
18.175 Lecture 21
![Page 57: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/57.jpg)
Outline
Recollections
General setup and basic properties
Recurrence and transience
18.175 Lecture 21
![Page 58: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/58.jpg)
Outline
Recollections
General setup and basic properties
Recurrence and transience
18.175 Lecture 21
![Page 59: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/59.jpg)
Query
I Interesting question: If A is an infinite probability transitionmatrix on a countable state space, what does the (infinite)matrix I + A + A2 + A3 + . . . = (I − A)−1 represent (if thesum converges)?
I Question: Does it describe the expected number of y hitswhen starting at x? Is there a similar interpretation for otherpower series?
I How about eA or eλA?
I Related to distribution after a Poisson random number ofsteps?
18.175 Lecture 21
![Page 60: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/60.jpg)
Query
I Interesting question: If A is an infinite probability transitionmatrix on a countable state space, what does the (infinite)matrix I + A + A2 + A3 + . . . = (I − A)−1 represent (if thesum converges)?
I Question: Does it describe the expected number of y hitswhen starting at x? Is there a similar interpretation for otherpower series?
I How about eA or eλA?
I Related to distribution after a Poisson random number ofsteps?
18.175 Lecture 21
![Page 61: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/61.jpg)
Query
I Interesting question: If A is an infinite probability transitionmatrix on a countable state space, what does the (infinite)matrix I + A + A2 + A3 + . . . = (I − A)−1 represent (if thesum converges)?
I Question: Does it describe the expected number of y hitswhen starting at x? Is there a similar interpretation for otherpower series?
I How about eA or eλA?
I Related to distribution after a Poisson random number ofsteps?
18.175 Lecture 21
![Page 62: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/62.jpg)
Query
I Interesting question: If A is an infinite probability transitionmatrix on a countable state space, what does the (infinite)matrix I + A + A2 + A3 + . . . = (I − A)−1 represent (if thesum converges)?
I Question: Does it describe the expected number of y hitswhen starting at x? Is there a similar interpretation for otherpower series?
I How about eA or eλA?
I Related to distribution after a Poisson random number ofsteps?
18.175 Lecture 21
![Page 63: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/63.jpg)
Recurrence
I Consider probability walk from y ever returns to y .
I If it’s 1, return to y infinitely often, else don’t. Call y arecurrent state if we return to y infinitely often.
18.175 Lecture 21
![Page 64: 18.175: Lecture 21 .1in More Markov chainsmath.mit.edu/~sheffield/2016175/Lecture21.pdf · 18.175 Lecture 21. Ergodic Markov chains I Say Markov chain is ergodic if some power of](https://reader036.fdocuments.us/reader036/viewer/2022071210/6021f0de2763a6445a629ff3/html5/thumbnails/64.jpg)
Recurrence
I Consider probability walk from y ever returns to y .
I If it’s 1, return to y infinitely often, else don’t. Call y arecurrent state if we return to y infinitely often.
18.175 Lecture 21