Post on 06-Aug-2020
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Lecture 2.2 Finite State Markov Chains
A. Banerji
Department of Economics
February 24, 2014
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Outline
Markov ChainsIntroductionMarginal DistributionsIdentities
Stability of Finite State MCsStationary DistributionsDobrushin Coefficient
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Stochastic Kernels
Finite State Space S = {x1, x2, ..., xN}Distribution on S. A function φ : S → < s.t. φ(x) ≥ 0, forall x ∈ S and
∑x∈S φ(x) = 1.
The set of all distributions on S, P(S), is the N-1dimensional unit simplex in <N .
DefinitionA stochastic kernel on S is a function p : S × S → [0,1]s.t.
∑y∈S p(x , y) = 1, for all x ∈ S.
For each x ∈ S, we call the corresponding distribution onS, p(x ,dy). For a finite state space S, we can write downthe N distributions in an N × N matrix.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Markov Chains
M = (p(x ,dy))x∈S
=
p(x1, x1) . . . p(x1, xN)...
......
p(xN , x1) . . . p(xN , xN)
DefinitionThe Markov Chain on S generated by stochastic kernel pand initial condition ψ ∈ P(S) is the sequence (Xt)
∞t=0 of
random variables defined by(i) X0 ∼ ψ(ii)For t = 0,1,2, ..., Xt+1 ∼ p(Xt ,dy)So if Xt = x , P(Xt+1 = y |Xt = x) = p(x , y). CalledMarkov -(p, ψ) chain. Discuss Hamilton(2005),Quah(1993).
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Marginal Distribution - Approximation
Let (Xt)∞t=0 be a Markov Chain on S generated by a
stochastic kernel p and initial condition ψ. The marginaldistribution ψt(y) ≡ P(Xt = y), for all y ∈ S.Approximating ψt(y) by Monte-Carlo. Draw Xt a largenumber of times and compute the relative frequency of y .Specifically:Draw X0 from ψ a large n number of times. Each of thesetimes:For k = 1, ..., t , draw Xk from p(xk−1,dy).For y ∈ S, ψt(y) ' 1
n∑n
i=1 1{X it =y}
Now do JS exercises 4.2.1, 4.2.2.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Marginal Distribution - Recursion
By the Law of Total Probability,P{Xt+1 = y} =
∑x∈S P{Xt+1 = y |Xt = x}P{Xt = x}
(The above just integrates out Xt from the joint (Xt+1,Xt)).That is,ψt+1(y) =
∑x∈S p(x , y)ψt(x) = (ψt(x))x∈S.(p(x , y))x∈S
Stacking these for all y ∈ S in one row,ψt+1 = (ψt+1(y))y∈S = ψtMBy t recursions, we getψt+1 = ψM t+1 †The probabilities of the states at t + 1 is a weightedaverage of the transitions p(x ,dy) (rows of M) weightedby the probabilities of the states at t .
ExampleQuah - Starting in extreme poverty (State 1), what is themarginal distribution after 10, 60 and 160 periods.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Powers of M
Let (pk (x , y))N×N ≡ Mk
Lemmapk (x , y) = P{Xt+k = y |Xt = x}
Proof.Let δx ∈ P(S) be the degenerate distribution that gives xwith probability 1.So P{Xt+k = y |Xt = x} = P{Xt+k = y |Xt ∼ δx}This is just the marginal distribution ψt+k (y) with initialcondition Xt ∼ δx . By recursion †,ψt+k = δxMk = pk (x ,dy).
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Expectation
Suppose Xt ∼ ψ ∈ P(S). So the marginal distribution ofXt+k , ψt+k = ψMk . So if h : S → <, the expectationE[h(Xt+k )|Xt ∼ ψ] =
∑y∈S ψMk (y)h(y) = ψMkh
where h ≡ (h(y))y∈S (we’ve taken an inner product).
ExampleIf ψ = δx , we haveE[h(Xt+K )|Xt = x ] =
∑y∈S pk (x , y)h(y) = δxMkh. This is
just the x th row of the matrix Mk multiplied by the vector h.For Hamilton(2005), let h = (1000,0,−1000) be profits ofa firm in the 3 states. What is expected profit 5 periodsfrom now, if we are currently in severe recession (state1)? Do JS 4.2.4-5.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Chapman-Kolmogorov Equation
The Equation:
pk+j(x , y) =∑z∈S
pk (x , z)pj(z, y)
Proof.Mk+j = MkM j . So the (x , y)th element of Mk+j is theinner product of the xth row of Mk and yth column ofM j .To go from state x to state y in k + j steps, we must go tostate z ∈ S in k steps, then from there to y in j steps. Forfixed z, multiply the 2 probabilities; then add over all(mutually exclusive) z ’s. In other standard notation,P{Xk+j = y |X0 = x} =
∑z∈S P{Xk+j = y |Xk = z}P{Xk =
z|X0 = x}
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Exercise
JS 4.2.6. In terms of sums,pk (x , y) =
∑z1∈S p(x , z1)
∑z2∈S p(z1, z2) . . .∑
zk−1∈S p(zk−2, zk−1)p(zk−1, y)
Proof.pk (x , y) is the sum of probabilities of all mutuallyexclusive outcome paths of type {xz1z2 . . . zk−1y}, whichequals∑
all {xz1z2...zk−1y} p(x , z1)p(z1, z2) . . . p(zk−1, y)=∑
all{xz1z2...zk−2} p(x , z1) . . .
p(zk−3, zk−2)∑
zk−1∈S p(zk−2, zk−1)p(zk−1, y)where we’ve fixed {xz1 . . . zk−2} and summed across thelast stretch of the paths ending at y . Working backwardsall the way we get∑
z1∈S p(x , z1)∑
z2∈S p(z1, z2)∑
z3∈S . . .∑zk−1∈S p(zk−2, zk−1)p(zk−1, y)
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Introduction
I Investigate the sequence (ψt) of marginaldistributions for Quah, as t grows large.
I (ψt) settles at some ψ∗, regardless of where we startI Global asymptotic stability of Markov Chains refers to
the settling down of the marginal distribution to aunique distribution, regardless of initial condition
I Known as ergodicity
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Dynamical System Corresponding to FSMC
I The marginal distributions of the Markov Process(Xt) with matrix M are (ψt) = (ψM t), ifX0 ∼ ψ ∈ P(S).
I Notice that M : P(S)→ P(S) (JS 4.3.1.). Indeed, forany ψ ∈ P(S), ψM =
∑x∈S ψ(x)p(x ,dy). So, for all
y ∈ S, the y th coordinate of ψM,ψM(y) =
∑x∈S ψ(x)p(x , y) > 0. Also,
∑y∈S ψM(y)
=∑
y∈S∑
x∈S ψ(x)p(x , y)=∑
x∈S ψ(x)∑
y∈S p(x , y) =∑
x∈S ψ(x) = 1. So,ψM ∈ P(S).Basically, ψM is a convex combination of points fromP(S) and therefore belongs to P(S).
I Impose the norm || ||1 and the corresponding metricd1 on P(S). Then (P(S),M) is a dynamical system,with ψt+1 = ψtM, t = 0,1,2, . . ..
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Stationary Distributions
DefinitionA distribution ψ∗ ∈ P(S) is stationary or invariant for M ifψ∗M = ψ∗. That is, ψ∗ is a fixed point of the dynamicalsystem (P(S),M).
TheoremEvery Markov chain on a finite state space has at leastone stationary distribution.
Proof.P(S) is compact and convex (it’s just the (N-1)dimensional unit simplex), and M is linear and hencecontinuous. So by Brouwer’s fixed point theorem, M hasa fixed point on P(S).Note: There could be many fixed points. e.g. JS 4.3.4.For the Markov matrix IN (the N × N identity matrix),every ψ ∈ P(S) is stationary.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Some Implications
LemmaM is d1-nonexpansive on P(S). That is, for allψ,ψ
′ ∈ P(S), d1(ψM, ψ′M) ≤ d1(ψ,ψ
′).
Proof.
||ψM − ψ′M||1 =∑y∈S
|ψM(y)− ψ′M(y)|
=∑
y∈S |∑
x∈S(ψ(x)− ψ′(x))p(x , y)|
≤∑
y∈S∑
x∈S |(ψ(x)− ψ′(x))p(x , y)|
=∑x∈S |ψ(x)− ψ
′(x)|
∑y∈S p(x , y) =
∑x∈S |ψ(x)− ψ
′(x)|
= ||ψ − ψ′ ||1.The inequality follows from the triangleinequality.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Computing Stationary Distributions
ψ ∈ P(S) is stationary or a fixed point iff ψ(IN −M) = 0i.e. (IN −M)TψT = 0. We can solve the system ofequations, and normalize ψT by dividing by its norm, sothat it lies in P(S). Alternatively: JS 4.3.5. Let
1N ≡ (1,1, . . . ,1) and 1N×N an N × N matrix of ones. If ψis a fixed point of M and ψ ∈ P(S), we have1N = ψ(IN −M + 1N×N). Indeed, since ψ ∈ P(S), so
ψ1N×N = 1N . So, ψ(IN −M) = 0, or ψ is a fixed point ofM. However, if ψ /∈ P(S), then it is not necessarily truethat 1N = ψ(IN −M + 1N×N).
So solving (IN −M + 1N×N)TψT = 1T
N works. (do JS4.3.6-7)
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
StabilityDefinitionThe dynamical system (P(S),M) is globally stable if ithas a unique stationary distribution (fixed point)ψ∗ ∈ P(S), and for all ψ ∈ P(S),d1(ψM t , ψ∗) ≡ ||ψM t − ψ∗||1 → 0, as t →∞.Need more than nonexpansiveness of M for stability.Stability fails for M = IN . Succeeds ‘best’ if p(x ,dy) isidentical for all x ∈ S (we then jump to the unique fixedpoint in a single step, from any ψ ∈ P(S)).
Example
M =
(0 11 0
)ψ∗ = (1/2,1/2) is the unique fixed point; for every otherψ = (ψ1,1− ψ1) 6= ψM = (1− ψ1, ψ1). This also showsthat (ψM t) oscillates with t , so the system is not globallystable.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Dobrushin Coefficient
DefinitionThe Dobrushin Coefficient of a stochastic kernel p is
α(p) ≡ min(x ,x ′ )∈S×S
∑y∈S
p(x , y) ∧ p(x′, y)
where a ∧ b ≡ min{a,b}Remarks.1. Hamilton and Quah. α(p) equals 0.029 forMH and 0 for MQ (see 1st and 5th rows of MQ). 2.α(p) ∈ [0,1], for all p. It equals 1 iff p(x ,dy) is identicalfor all x ∈ S. It equals 0 for IN and the periodic kernel onthe previous slide. 3. α(p) > 0 iff for every pair(x , x
′) ∈ S × S, p(x ,dy) and p(x
′,dy) overlap (assign
positive probability to at least one common state y ). Fromany 2 states then, there is positive probability that thechains will meet next period.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Dobrushin Coefficient and Stability
TheoremLet p be a stochastic kernel with Markov matrix M. Thenfor every φ, ψ ∈ P(S),
||φM − ψM||1 ≤ (1− α(p))||φ− ψ||1Moreover this bound is tight; for λ < (1− α(p)), thereexists a pair φ, ψ which violates the ≤ inequality.The proof consists of 3 steps/lemmas.
LemmaJS C.2.1. Let φ, ψ ∈ P(S) and h : S → <+. Then
|∑x∈S
h(x)φ(x)−∑x∈S
h(x)ψ(x)| ≤ 12
supx ,x ′|h(x)−h(x
′)|.||φ−ψ||1
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Stability 2
JS C.2.1. provides an upper bound for the (absolute)difference of expectation h under φ and ψ. See proof inStachurski (appendix).
LemmaC.2.2.||φM −ψM||1 ≤ 1
2 supx ,x ′ ||p(x ,dy)− p(x′,dy)||1.||φ−ψ||1
Proof.See Stachurski (appendix). The inequality looks similar toC.2.1. Exercise 4.3.2. implies||φM − ψM||1 = 2 supA⊂S |φM(A)− ψM(A)|. Weintroduce the function h used in C.2.1. by noting that|φM(A)− ψM(A)| =|∑
x∈S P(x ,A)φ(x)−∑
x∈S P(x ,A)ψ(x)|, whereP(x ,A) =
∑y∈A p(x , y).
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Stability 3To prove the first claim of the theorem, we now show that
12
supx ,x ′||p(x ,dy)−p(x
′,dy)||1 = 1− inf
x ,x ′
∑y∈S
p(x , y)∧p(x′, y)
It suffices to show that for every x , x′,
12||p(x ,dy)− p(x
′,dy)||1 = 1−
∑y∈S
p(x , y) ∧ p(x′, y)
This is true for any pair of distributions, as below.
LemmaC.2.3. For every pair µ, ν ∈ P(S) we have
12||µ− ν|| = 1−
∑y∈S
µ(y) ∧ ν(y)
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Stability 4
To show that the bound is tight, note that 1− α(p)= 1
2 supx ,x ′ ||p(x ,dy)− p(x′,dy)||1
= sup {x 6= x′} ||p(x ,dy)−p(x
′,dy)||1
||δx−δx′ ||1≤ supµ6=ν
|µM−νM||1|µ−ν||1
The second equality holds since ||δx − δx ′ ||1 = 2. Thefinal inequality holds because the set of degeneratedistributions like δx , δx ′ is a subset of the set of alldistributions. More simply, just put M = IN (so α(p) = 0).Now for the main theorem.
TheoremLet p be a stochastic kernel on a finite set S, and M thecorresponding Markov matrix. The dynamical system(P(S),M) is globally stable iff there exists a t ∈ N s.t.α(pt) > 0.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Stability 5
Proof.M is nonexpansive. From the earlier theorem, we knowthat since α(pt) > 0, (P(S),M t) is globally stable. So bylemma 4.1.1., (P(S),M) is globally stable.Conversely, suppose (P(S),M) is globally stable. Sothere is a unique stationary distribution ψ∗, andψM t → ψ∗ for all ψ ∈ P(S). In particular,δxM t = pt(x ,dy)→ ψ∗, for every x ∈ S. So for all x ∈ S,pt(x , y)→ ψ∗(y), for all y ∈ S. Since ψ∗ is a distribution,there is at least one y ∈ S s.t. ψ∗(y) > 0. So for this y ,we have from the convergence that for t large enough,pt(x , y) > 0, for all x ∈ S. Thus there exists t such that allrows pt(x ,dy) of M t overlap at this y ; thus the Dobrushincoefficient α(pt) > 0.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
ExercisesTheorem shows:for every pair (x , x
′) of states, pt(x ,dy)
and pt(x′,dy) overlap. Starting at any 2 different points
today, the chains meet with positive probability t periodslater. Extreme form: pt(x ,dy) same for all x orconvergence in t periods.
I α(p) > 0 for Hamilton’s matrix but zero for Quah’smatrix. However, for the 23rd iterate M23
Q reported byQuah, α(p23) > 0. So (P(S),MQ) is globally stable.
I JS 4.3.20. Code to calculate α(pt), t = 1,2, . . . ,T fora given Markov matrix M, stopping at the first t s.t.α(pt) > 0. Show that this t = 2 for Quah’s matrix.
I (s,S) (or (q,Q)) inventory dynamics. A firm withinventory Xt at the start of period t , has the option ofordering inventory up to its maximum storingcapacity Q. At the end of period t , demand Dt+1 isobserved (all non-negative integers). The firm meetsdemand up to its current stock level; remaininginventory is carried over to next period.
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
Inventory Dynamics exercise
(Dt)t≥1 is an iid sequence of random variables takingnonnegative integer values according to distributionb(d) ≡ P{Dt = d} = (1/2)d+1.The firm follows a stationary policy: If Xt ≤ q, orderinventory to top up to equal Q; otherwise, order noinventory (the choice of q is the firm’s predecided policychoice).So,
Xt+1 =
{max{Q − Dt+1,0} if Xt ≤ qmax{Xt − Dt+1,0} if Xt > q
Let S = {0,1, ...,Q}. What is the stochastic kernelMq = (p(x , y)) corresponding to restocking policy q?
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
(q,Q) DynamicsLet x ≤ q. Then
Xt+1 =
{Q − i with Pr(1/2)i+1, i = 0,1, ...,Q − 1
0 with Pr(1/2)Q
Let x > q. Then
Xt+1 =
{x − i with Pr(1/2)i+1, i = 0,1, ..., x − 1
0 with Pr(1/2)x
Mq =
(1/2)Q (1/2)Q (1/2)Q−1 . . . . . . (1/2). . . . . . . . . . . . . . . . . .
(1/2)Q (1/2)Q (1/2)Q−1 . . . . . . (1/2). . . . . . . . .
(1/2)x (1/2)x (1/2)x−1 . . . (1/2) 0. . . . . . . . .
(1/2)Q (1/2)Q (1/2)Q−1 . . . . . . (1/2)
StochasticDynamics
A. Banerji
Markov ChainsIntroduction
Marginal Distributions
Other Identities
Stability of FiniteState MCsStationary Distributions
Dobrushin Coefficient
(q,Q) Dynamics cont
Staring at Mq, we see that regardless of q (and Q),α(p) > 0. So (P(S),Mq) is globally stable.
JS 4.3.23. Compute the stationary distribution when(q,Q) = (2,5).JS 4.3.24. Suppose Q = 20, and the fixed cost ofordering inventory in any period is 0.1. The firm buys theproduct at zero cost per unit and sells at USD 1 per unit.For each q ∈ {0,1,2, ...,20}, evaluate the stationarydistribution ψ∗q, and evaluate the firm’s expected perperiod profit at this stationary distribution (i.e. computethe firm’s long run average profits with restocking policyq). Show that this profit is maximized at q = 7.