Chapter 2

Continuous time Markov chains

As before we assume that we have a finite or countable statespace I, but now the Markov chainsX = {X(t) : t ≥ 0} have a continuous time parameter t ∈ [0,∞). To avoid technical difficulties wewill always assume that X changes its state finitely often in any finite time interval.

2.1 Q-Matrices

In continuous time there are no smallest time steps and hence we cannot speak about one-step tran-sition matrices any more. If we are in state j ∈ I at time t, then we can ask for the probability ofbeing in a different state k ∈ I at time t + h,

f(h) = IP{X(t + h) = k |X(t) = j}.

We are interested in small time steps, i.e. small values of h > 0. Clearly, f(0) = 0. Assuming thatf is differentiable at 0, the most interesting information we can obtain about f at the origin is itsderivative f ′(0), which we call qjk.

limh↓0

IP{X(t + h) = k |X(t) = j}

h= lim

h↓0

f(h) − f(0)

h − 0= f ′(0) = qjk.

We can write this as

IP{X(t + h) = k |X(t) = j} = qjkh + o(h).

Here o(h) is a convenient abbreviation for any function with the property that limh↓0 o(h)/h = 0, wesay that the function is of smaller order than h. An advantage of this notation is that the actualfunction o might be a different one in each line, but we do not need to invent a new name for eachoccurrence.

Again we have the Markov property, which states that if we know the state X(t) then all additionalinformation about X at times prior to t is irrelevant for the future: For all k 6= j, t0 < t1 < . . . < tn < tand x0, . . . , xn ∈ I,

IP{X(t + h) = k |X(t) = j, X(ti) = xi∀i} = IP{X(t + h) = k |X(t) = j} = qjkh + o(h).

1

From this we get, for every j ∈ I,

IP{X(t + h) = j |X(t) = j} = 1 −∑

k 6=j

qjkh + o(h) = 1 + qjjh + o(h),

if we define qjj = −∑

k 6=j qjk. As above we have qjj = f ′(0) for f(h) = IP{X(t + h) = j |X(t) = j},but now f(0) = 1.

We can enter this information into a matrix Q = (qij : i, j ∈ I), which contains all the informationabout the transitions of the Markov chain X. This matrix is called the Q-matrix of the Markov chain.Its necessary properties are

• all off-diagonal entries qij , i 6= j, are non-negative,

• all diagonal entries qii are nonpositive and,

• the sum over the entries in each row is zero (!).

We say that qij gives the rate at which we try to enter state j when we are in state i, or the jump

intensity from i to j.

Example 2.1: The Poisson process

Certain random occurrences (for example claim times to an insurance company, arrival times of cus-tomers in a shop,. . . ) happen randomly in such a way that

• the probability of one occurrence in the time interval (t, t + h) is λh + o(h), the probability ofmore than one occurrence is o(h).

• occurrences in the time interval (t, t + h) are independent of what happened before time t.

Suppose now that X(t) is the number of occurrences by time t. Then X is a continuous time Markovchain with statespace I = {0, 1, 2, . . .}. Because, by our assumptions,

IP{X(t + h) = k |X(t) = j} =

λh + o(h) if k = j + 1,0 + o(h) if k > j + 1,0 if k < j,

we get the Q-matrix

Q =

−λ λ 0 0 . . .0 −λ λ 0 . . .0 0 −λ λ 0 . . .

0 . . . 0. . .

. . .. . .

.

This process is called a Poisson process with rate λ.

Reason for this name: An analysis argument shows that, if X(0) = 0, then

IP{X(t) = n} = e−λt (λt)n

n!for n = 0, 1, 2, . . . ,

i.e. X(t) is Poisson distributed with parameter λt.

Example 2.2: The pure birth-process

Suppose we have a population of (immortal) animals reproducing in such a way that, independent ofwhat happened before time t and of what happens to the other animals, in the interval (t, t + h) eachanimal

2

• gives birth to 1 child with probability λh + o(h),

• gives birth to more than 1 child with probability o(h),

• gives birth to 0 children with probability 1 − λh + o(h).

Let p = λh + o(h) and q = 1 − p and X(t) the number of animals at time t. Then,

IP{X(t + h) = n |X(t) = n} = IP{ no births in (t, t + h) |X(t) = n}

= qn = [1 − λh + o(h)]n

= 1 − nλh + o(h),

and

IP{X(t + h) = n + 1 |X(t) = n} = IP{ one birth in (t, t + h) |X(t) = n}

= npqn−1 = nλh [1 − λh + o(h)]n−1

= nλh + o(h),

and finally, for k > 1,

IP{X(t + h) = n + k |X(t) = n} = IP{k births in (t, t + h) |X(t) = n}

=

(n

k

)pkqn−k + o(h) =

(n

k

)λkhk [1 − λh + o(h)]n−k + o(h)

= o(h).

Hence, X is a continuous time Markov chain with Q-matrix

Q =

−λ λ 0 0 . . .0 −2λ 2λ 0 . . .0 0 −3λ 3λ 0 . . .

0 . . . 0. . .

. . .. . .

.

An analysis argument shows that, if X(0) = 1, then

IP{X(t) = n} = e−λt(1 − e−λt)n−1 for n = 1, 2, 3, . . . ,

i.e. X(t) is geometrically distributed with parameter e−λt.

2.1.1 Construction of the Markov chain

Given the Q-matrix one can construct the paths of a continuous time Markov chain as follows. Supposethe chain starts in a fixed state X0 = i for i ∈ I. Let

J = min{t : Xt 6= X0}

be the first jump time of X (we always assume that the minimum exists).

3

Theorem 2.1 Under the law IPi of the Markov chain started in X0 = i the random variables J and

X(J) are independent. The distribution of J is exponential with rate qi :=∑

j 6=i qij, which means

IPi{J > t} = e−qit for t ≥ 0.

Moreover,

IPi{X(J) = j} =qij

qi,

and the chain starts afresh at time J .

Let J1, J2, J3, . . . be the times between successive jumps. Then

Yn = X(J1 + · · · + Jn)

defines a discrete time Markov chain with one-step transition matrix P given by

pij =

{qij

qiif i 6= j,

0 if i = j.

Given Yn = i the next waiting time Jn+1 is exponentially distributed with rate qi and independent ofY1, . . . , Yn−1 and J1, . . . , Jn.

2.1.2 Exponential times

Why does the exponential distribution play a special role for continuous Markov chains?

Recall the Markov property in the following way: Suppose X(0) = i and let J be the time before thefirst jump and t, h > 0. Then, using the Markov property in the second step,

IP{J > t + h | J > t} = IP{J > t + h | J > t, X(t) = i} = IP{J > t + h |X(t) = i} = IP{J > h}.

Hence the time J we have to wait for a jump satisfies the lack of memory property : if you have waitedfor t time units and no jump has occurred, the remaining waiting time has the same distribution asthe original waiting time. This is sometimes called the waiting time paradox or constant failure rate

property.

The only distribution with the lack of memory property is the exponential distribution. Check that,for IP{J > x} = e−µx, we have

IP{J > t + h | J > t} =e−µ(t+h)

e−µt= e−µh = IP{J > h}.

Another important property of the exponential distribution is the following:

Theorem 2.2 If S and T are independent exponentially distributed random variables with rate α resp.

β, then their minimum S ∧T is also exponentially distributed with rate α + β and it is independent of

the event {S ∧ T = S}. Moreover,

IP{S ∧ T = S} =α

α + βand IP{S ∧ T = T} =

β

α + β.

Prove this as an exercise, see Sheet 9!

4

Example 2.3: The M/M/1 queue.

Suppose customers arrive according to a Poisson process with of rate λ at a single server. Eachcustomer requires an independent random service time, which is exponential with mean 1/µ (i.e. withrate µ). Let X(t) be the number of people in the queue (including people currently being served).Then X is a continuous-time Markov chain with

Q =

−λ λ 0 0 0 . . .µ −(λ + µ) λ 0 0 . . .0 µ −(λ + µ) λ 0 . . .

0 0. . .

. . .. . . . . .

.

The previous theorem says: If there is at least one person in the queue, the distribution of the jumptime (i.e. the time until the queue changes) is exponential with rate λ + µ and the probability that atthe jump time the queue is getting shorter is µ/(λ + µ).

2.2 Kolmogorov’s equations and transition matrix functions

LetP (t) =

(pij(t), i, j ∈ I, t ≥ 0

)given by pij(t) = IPi{X(t) = j},

be the transition matrix function of the Markov chain. From P one can get the full information aboutthe law of the Markov chain, for 0 < t1 < . . . < tn and j1, . . . , jn ∈ I,

IPi{X(t1) = j1, . . . , X(tn) = jn} = pij1(t1)pj1j2(t2 − t1) · · · pjn−1jn(tn − tn−1).

P has the following properties,

(a) pij(t) ≥ 0 and∑

j∈I

pij(t) = 1 for all i ∈ I. Also

limt↓0

pij(t) =

{1 if i = j,0 otherwise.

(b)∑

k∈I

pik(t)pkj(s) = pij(t + s) for all i, j ∈ I and s, t ≥ 0.

The equation in (b) is called the Chapman-Kolmogorov equation. It is easily proved,

IPi{X(t + s) = j} =∑

k∈I

IPi{X(t) = k, X(t + s) = j}

=∑

k∈I

IPi{X(t) = k}IPi{X(t + s) = j |X(t) = k}

=∑

k∈I

IPi{X(t) = k}IPk{X(s) = j}.

Note that, by definition pjk(h) = qjkh + o(h) for j 6= k, and hence

p′jk(0) = qjk and p′jj(0) = qjj .

5

From this we can see how the Q-matrix is obtainable from the transition matrix function. The converseoperation is more involved, we restrict attention to the case of finite statespace I.

Consider P (t) as a matrix, then the Chapman-Kolmogorov equation can be written as

P (t + s) = P (t)P (s) for all s, t ≥ 0.

One can differentiate this equation with resp to s or t and gets

P ′(t) = QP (t) and P ′(t) = P (t)Q.

These equations are called the Kolmogorov backward resp. Kolmogorov forward equations. We alsoknow P (0) = I, the identity matrix. This matrix-valued differential equation has a unique solutionP (t), t ≥ 0.

For the case of a scalar function p(t) and a scalar q instead of the matrix-valued function P and matrixQ the corresponding equation

p′(t) = qp(t), p(0) = 1,

would have the unique solution p(t) = etq. In the matrix case one can argue similarly, defining

eQt =∞∑

k=0

Qktk

k!,

with Q0 = I. Then we get the transition matrix function P (t) = eQt which satisfies the Kolmogorovforward and backward equations.

2.3 Resolvents

Let Q be the Q-matrix of the Markov chain X and P (t), t ≥ 0 the transition matrix function.

2.3.1 Laplace transforms

A function f : [0,∞) → IR is called good if f is continuous and either bounded (there exists K > 0such that |f(t)| ≤ K for all t) or integrable (

∫∞0 |f(t)| dt < ∞) or both. If f is good we can associate

the Laplace transform f : (0,∞) → IR which is defined by

f(λ) =

∫ ∞

0e−λtf(t) dt.

An important example is the case f(t) = e−αt. Then

f(λ) =

∫ ∞

0e−λte−αt dt =

1

λ + α.

An important result about Laplace transforms is the following:

Theorem 2.3 (Uniqueness Theorem) If f and g are good functions on [0,∞) and f(λ) = g(λ)for all λ > 0. Then f = g.

This implies that we can invert Laplace transforms uniquely, at least when restricting attention togood functions.

6

2.3.2 Resolvents

The basic idea here is to calculate the exponential P (t) = eQt of the Q-matrix using the Laplacetransform. Let λ > 0 and argue that

R(λ) :=

∫ ∞

0e−λtP (t) dt

=

∫ ∞

0e−λteQt dt =

∫ ∞

0e−(λI−Q)t dt

= (λI − Q)−1.

Here the integrals over matrix-valued functions are understood componentwise, and the last step isdone by analogy with the real-valued situation, but can be made rigorous. Note that the inverse ofthe matrix λI − Q can be calculated for all values λ which are not an eigenvalue of Q. Recall thatthe inverse fails to exist if and only if the characteristic polynomial det(λI − Q) is zero, which is alsothe criterion for λ to be an eigenvalue of Q.

Now R(λ) = (λI − Q)−1, if it exists, can be calculated and P (t) can be recovered by inverting theLaplace transform. The matrix function R(λ) is called the resolvent of Q (or of X). Its componentsare the Laplace transforms of the functions pij ,

rij(λ) :=

∫ ∞

0e−λtpij(t) dt = pij(λ).

Resolvents also have a probabilistic interpretation: Suppose we have an alarm-clock which rings,independently of the Markov chain, at a random time A, which is exponentially distributed withrate λ, i.e.

IP{A > t} = e−λt, IP{A ∈ (t, t + dt)} = λe−λt dt.

What is the state of the Markov chain when the alarm clock rings? The probability of being in statej is

IPi{X(A) = j} =

∫ ∞

0IPi{X(t) = j, A ∈ (t, t + dt)} =

∫ ∞

0pij(t)λe−λt dt = λrij(λ),

hence

IPi{X(A) = j} = λrij(λ) for A independent of X, exponential with rate λ.

We now discuss the use of resolvents in calculations for continuous-time Markov chains using thefollowing problem:

Problem: Consider a continuous-time Markov chain X with state space I := {A, B, C} and transi-tions between the states described as follows:

• When currently in state A at time t, in the next time interval (t, t + h) of small length h andindependent of the past behaviour of the chain, with

– probability h + o(h) the chain jumps into state B,

– probability 2h + o(h) the chain jumps into state C,

– probability 1 − 3h + o(h) the chain remains in state A.

7

• Whilst in state B, the chain tries to enter state C at rate 2. The chain cannot jump into stateA directly from state B.

• On entering state C, the chain remains there for an independent exponential amount of time ofrate 3 before jumping. When the jump occurs, with probability 2/3 it is into state A, and withprobability 1/3 the jump is to state B.

Questions:

(a) Give the Q-matrix of the continuous-time Markov chain X.

(b) If the Markov chain is initially in state A, that is X(0) = A, what is the probability that theMarkov chain is still in state A at time t, that is, IPA{X(t) = A} ?

(c) Starting from state C at time 0, show that the probability X is in state B at time t is given by

1

3−

1

3e−3t for t ≥ 0.

(d) Initially, X is in state C. Find the distribution for the position of X after an independentexponential time, T , of rate 4 has elapsed. In particular, you should find

IPC{X(T ) = B} =1

7.

(e) Given that X starts in state C, find the probability that X enters state A at some point beforetime t. [See Section 2.3.3 for solution.]

(f) What is the probability that we have reached state C at some point prior to time U , where Uis an independent exponential random variable of rate µ = 2.5, given that the chain is in stateB at time zero? [See Section 2.3.3 for solution.]

The method to obtain (a) should be familiar to you by now. For the jumps from state C we useTheorem 2.2: Recall qCA is the rate of jumping from C to A and qCB is the rate of jumping from C toB. We are given 3 = qCA + qCB, the rate of leaving state C. The probability that the first jump goesto state A is, again by Theorem 2.2, qCA/(qCA + qCB), which is given as 2/3. Hence we get qCA = 2and qCB = 1. Altogether,

Q =

−3 1 20 −2 22 1 −3

.

We now solve (b) with the resolvent method. Recall

pAA(t) = IPA{X(t) = A}, and rAA(λ) =

∫ ∞

0e−λtpAA(t) dt = pAA(λ).

8

As R(λ) = (λI − Q)−1 we start by inverting

λI − Q =

λ + 3 −1 −20 λ + 2 −2−2 −1 λ + 3

.

First,

det(λI − Q) = (λ + 2) det

(λ + 3 −2−2 λ + 3

)+ 2 det

(λ + 3 −1−2 −1

)

= λ(λ + 3)(λ + 5).

(Side remark: λ is always a factor in this determinant, because Q is a singular matrix. Recall thatthe sum over the rows is zero!) Now the inverse of a matrix A is given by

(A−1)ij =(−1)i+j

detAdet(Mji),

where Mji is the matrix with row j and column i removed. Hence, in our example, the entry inposition AA is obtained by

rAA(λ) = (λI − Q)−1AA =

det

(λ + 2 −2−1 λ + 3

)

det(λI − Q)=

λ2 + 5λ + 4

λ(λ + 3)(λ + 5).

It remains to invert the Laplace transform. For this purpose we need to form partial fractions. Solving

λ2 + 5λ + 4 = α(λ + 3)(λ + 5) + βλ(λ + 5) + γλ(λ + 3)

gives α = 4/15 (plug λ = 0) and β = 1/3 (plug λ = −3) and γ = 2/5 (plug λ = −5). Hence

rAA(λ) =4

15λ+

1

3(λ + 3)+

2

5(λ + 5).

Now the inverse Laplace transform of 1/(λ + β) is e−βt. We thus get

pAA(t) =4

15+

1

3e−3t +

2

15e−5t,

and note that pAA(0) = 1 as it should be!

Now solve part (c). To find IPC{X(t) = B} consider

rCB(λ) =

∫ ∞

0e−λtpCB(t) dt = pCB(t).

We need to find the CB coefficient of the matrix (λI − Q)−1, recalling the inversion formula we get

rCB(λ) =−det

(λ + 3 −1−2 −1

)

λ(λ + 3)(λ + 5)=

λ + 5

λ(λ + 3)(λ + 5)=

1

λ(λ + 3)=

1

3

( 1

λ−

1

λ + 3

),

9

and inverting Laplace transforms gives

IPC{X(t) = B} = pCB(t) =1

3−

1

3e−3t.

Suppose for part (d) that T is exponentially distributed with rate λ and independent of the chain.Recall,

IPi{X(T ) = j} = λrij(λ),

and by matrix inversion,

rCA(λ) =det

(0 λ + 2−2 −1

)

det(λI − Q)=

2(λ + 2)

λ(λ + 3)(λ + 5).

As T is exponential with parameter λ = 4 we get (no Laplace inversion needed!)

IPC{X(T ) = A} =2(4 + 2)

(4 + 3)(4 + 5)=

12

63=

4

21.

Similarly,

rCB(λ) =−det

(λ + 3 −1−2 −1

)

det(λI − Q)=

λ + 5

λ(λ + 3)(λ + 5),

and

rCC(λ) =det

(λ + 3 −1

0 λ + 2

)

det(λI − Q)=

(λ + 3)(λ + 2)

λ(λ + 3)(λ + 5).

Plugging λ = 4 we get

IPC{X(T ) = B} = 4rCB(4) =1

7and IPC{X(T ) = C} =

4 + 2

4 + 5=

2

3.

At this point it is good to check that the three values add up to one and thus define a probabilitydistribution on the statespace I = {A, B, C }.

2.3.3 First hitting times

Let Tj = inf{t > 0 : Xt = j} be the first positive time we are in state j. Define, for i, j ∈ I,

Fij(t) = IPi{Tj ≤ t} = IP{Tj ≤ t |X(0) = i},

then Fij(t) is the probability that we have entered state j at some time prior to time t, if we start thechain in i. We have the first hitting time matrix function,

F (t) =(Fij(t) : i, j ∈ I, t ≥ 0

),

with F (0) = I and

Fij(t) =

∫ t

0fij(s) ds =

∫ t

0IPi{Tj ∈ ds},

10

where fij(t) is the first hitting time density. The matrix functions f = (fij(t), t ≥ 0) and P are linkedby the integral equation

pij(t) =

∫ t

0fij(s)pjj(t − s) ds, for i, j ∈ I and t ≥ 0. (2.3.1)

This can be checked as follows, use the strong Markov property, which is the fact that the chain startsafresh at time Tj in state j and evolves independently of everything that happened before time Tj toget

pij(t) = IPi{Xt = j} = IPi{Xt = j; Tj ≤ t} =

∫ t

0IPi{Xt = j, Tj ∈ ds}

=

∫ t

0IPi{Tj ∈ ds}IPj{Xt−s = j} =

∫ t

0fij(s)pjj(t − s) ds.

Given two good functions f, g : [0,∞) → IR we can define the convolution function f ∗ g by

f ∗ g(t) =

∫ t

0f(s)g(t − s) ds,

and check by substituting u = t− s that f ∗ g = g ∗ f . We can rewrite the integral equation (2.3.1) as

pij(t) = fij ∗ pjj(t).

In order to solve this equation for the unknown fij we use Laplace transforms again.

Theorem 2.4 (Convolution Theorem) If f, g are good functions, then f ∗ g(λ) = f(λ)g(λ).

To make this plausible look at the example of two functions f(t) = e−αt and g(t) = e−βt. Then

f ∗ g(t) =

∫ t

0f(s)g(t − s) ds = e−βt

∫ t

0e−(α−β)s ds =

e−βt − e−αt

α − β.

Then

f ∗ g(λ) =

∫ ∞

0e−λs(f ∗ g)(s) ds =

1

α − β

{1

λ + β−

1

λ + α

}

=1

λ + α

1

λ + β= f(λ)g(λ).

The convolution theorem applied to the integral equation (2.3.1) gives a formula for the Laplacetransform of the first hitting time density

fij(λ) =rij(λ)

rjj(λ)for i, j ∈ I and λ > 0. (2.3.2)

We use this now to given an answer to (e) in our problem. We are looking for

IPC{TA ≤ t} = FCA(t) =

∫ t

0fCA(s) ds.

11

We first find the Laplace transform fCA of fCA, using rCA and rAA,

fCA =det

(0 λ + 2−2 −1

)

det

(λ + 2 −2−1 λ + 3

) =2(λ + 2)

λ2 + 5λ + 4,

and by partial fractions

fCA =2

3(λ + 1)+

4

3(λ + 4).

In this form, the Laplace transform can be inverted, which gives

fCA(t) =2

3e−t +

4

3e−4t for t ≥ 0.

Now, by integration, we get

IPC{TA ≤ t} =

∫ t

0fCA(s) ds

=2

3

∫ t

0e−s ds +

4

3

∫ t

0e−4sds

= 1 −2

3e−t −

1

3e−4t.

Check that IPC{TA ≤ 0} = 0, as expected.

As in the case of rij = pij the Laplace transforms fij also admit a direct probabilistic interpretationwith the help of an alarm clock, which goes off at a random time A, which is independent of theMarkov chain X and exponentially distributed with rate λ. Indeed, recalling that λe−xλ, x > 0 is thedensity of A and fij is the density of Tj ,

IPi{Tj ≤ A} =

∫ ∞

0IPi{Tj ≤ a}λe−aλ da

=

∫ ∞

0

∫ a

0fij(s) dsλe−aλ da

=

∫ ∞

0fij(s)

∫ ∞

sλe−aλ da ds

=

∫ ∞

0fij(s)e

−sλ ds

= fij(λ).

Altogether,

fij(λ) = IPi{Tj < A} for i, j ∈ I and λ > 0.

We use this now to given an answer to (f) in our problem. The question asks for IPB{TC < U}where U is exponential with rate µ = 2.5 and independent of the Markov chain. We now know that

IPB{TC < U} = fBC(µ) =rBC(µ)

rCC(µ)=

−det

(µ + 3 −2

0 −2

)

det

(µ + 3 −1

0 µ + 2

) =2(µ + 3)

(µ + 3)(µ + 2)=

2

µ + 2,

12

and plugging in µ = 2.5 gives IPB{TC < U} = 49 .

Example: A machine can be in three states A, B, C and the transition between the states is givenby a continuous-time Markov chain with Q-matrix

Q =

−3 1 23 −3 01 2 −3

.

At time zero the machine is in state A. Suppose a supervisor arrives for inspection at the site of themachine at an independent exponential random time T , with rate λ = 2. What is the probability thathe has arrived when the machine is in state C for the first time?

We write TC = inf{t > 0 : X(t) = C} for the first entry and SC = inf{t > TC : X(t) 6= C} for thefirst exit time from C. Then the required probability is IPA{TC < T, SC > T}. Note that J = SC −TC

is the length of time spent by the chain in state C during the first visit. This time is exponentiallydistributed with rate qCA + qCB = 3 and we get

IPA{TC < T, SC > T} = IPA{TC < T}IPA{SC > T |TC < T}

= IPA{TC < T}IPA{T − TC < J |T > TC}.

By the lack of memory property, given T > TC the law of T = T − TC is exponential with rate λ = 2again. Hence

IPA{TC < T, SC > T} = IPA{TC < T}IPC{T < J},

where T is an independent exponential random variable with rate λ = 2. The right hand side can becalculated. We write down

λI − Q =

λ + 3 −1 −2−3 λ + 3 0−1 −2 λ + 3

,

and derive

IPA{TC < T} = fAC(λ) =rAC(λ)

rCC(λ)=

det

(−1 −2

λ + 3 0

)

det

(λ + 3 −1−3 λ + 3

) =2(λ + 3)

λ2 + 6λ + 6,

and, by Theorem 2.2,

IPC{J > T} =λ

λ + 3

recalling that J and T are independent exponentially distributed with rates 3 resp. λ = 2. Hence,

IPA{TC < T, SC > T} =2λ

λ2 + 6λ + 6=

4

22=

2

11.

2.4 Long-term behaviour and invariant distribution

We now look at the long-term behaviour of continuous time Markov chains. The notions of anirreducible and recurrent Markov chain can be defined in the same manner as in the discrete case:

13

• A chain is irreducible if one can get from any state to any other state in finite time, and

• an irreducible chain is recurrent if the probability that we return to this state in finite time isone.

We assume in this chapter that our chain satisfies our usual conditions, and is irreducible.

2.4.1 The invariant distribution

Recall that in the discrete time case the stationary distribution π on I was given as the solution ofthe equation πP = π and, by iteration,

π = πP = πP 2 = πP 3 = · · · = πPn for all n ≥ 0.

Hence we would expect an invariant distribution π of a continuous time Markov chain with transitionmatrix function (P (t) : t ≥ 0) to satisfy

πP (t) = π for all t ≥ 0.

If the statespace is finite, we can differentiate with respect to time to get πP ′(t) = 0. Setting t = 0and recalling P ′(0) = Q we get

πQ = 0.

The two boxed statements can be shown to be equivalent in the general case, see Norris Theorem 3.5.5.Any probability distribution π satisfying πQ = 0 is called an invariant distribution, or sometimesequilibrium or stationary distribution.

Theorem 2.5 Suppose (X(t) : t ≥ 0) satisfies our assumptions and π solves

∑

i∈I

πiqij = 0, πj > 0 for all j ∈ I, and∑

i∈I

πi = 1,

then,

limt→∞

pij(t) = πj for all i ∈ I,

and if Vj(t) =∫ t0 1{X(s)=j} ds is the time spent in state j up to time t, we have

limt→∞

Vj(t)

t= πj with probability one.

This theorem is analogous to (parts of) the ergodic theorem in the discrete time case. Note that italso implies that the invariant

14

2.4.2 Symmetrisability

Suppose m = (mi : i ∈ I) satisfies mi > 0. We say that Q is m-symmetrisable if we can find mi > 0with

miqij = mjqji for all i, j ∈ I

These are the detailed balance equations.

Theorem 2.6 If we find an m solving the detailed balance equations such that M =∑

i∈I mi < ∞,

then πi = mi/M defines the unique invariant distribution.

For the proof note that, for all j ∈ I,

(πQ)j =1

M

∑

i∈I

miqij =1

M

∑

i∈I

mjqji = 0.

Note that, as in the discrete case, the matrix Q may not be symmetrisable, but the invariant distri-bution may still exist. Then the invariant distribution may be found using generating functions.

Example: The M/M/1 queue.

A single server has a service rate µ, customers arrive individually at a rate λ. Let X(t) be the numberof cutomers in the queue (including the customer currently served) and I = {0, 1, 2, . . .}. The Q-matrixis given by

Q =

−λ λ 0 . . .µ −λ − µ λ 0 . . .0 µ −λ − µ λ 0 . . .

. . . 0. . .

. . .. . . 0

.

We first try to find the invariant distribution using symmetrisability. For this purpose we have tosolve the detailed balance equations miqij = mjqji, explicitly

m0λ = m1µ ⇒ m1 =λ

µm0,

m1λ = m2µ ⇒ m2 =λ

µm1,

mnλ = mn+1µ ⇒ mn+1 =λ

µmn,

for all n ≥ 2. Hence denoting by ρ = λ/µ the traffic intensity we have mi = ρim0. If ρ < 1, then

M =∞∑

i=0

mi = m0

∞∑

i=0

ρi =m0

1 − ρ< ∞,

Hence get that

πi =mi

M= (1 − ρ)ρi for all i ≥ 0.

In other words, if ρ < 1 the invariant distribution is the geometric distribution with parameter ρ.Over a long time range the mean queue length is

∞∑

i=1

iπi =∞∑

i=1

(1 − ρ)iρi = (1 − ρ)ρ( ∞∑

i=1

ρi)′

=ρ

1 − ρ.

15

PS: In the case ρ > 1 the chain is not recurrent, in the case ρ = 1 the chain is null recurrent and noinvariant distribution exists.

As a (more widely applicable) alternative one can also solve the equation πQ = 0 using the generatingfunction

π(s) :=∞∑

n=0

snπn = π0 + π1s + π2s2 + · · · .

We write out the equation πQ = 0 as

−λπ0 + µπ1 = 0

λπ0 − (λ + µ)π1 + µπ2 = 0

λπ1 − (λ + µ)π2 + µπ3 = 0,

and so on. Multiplying the first equation by s, the second by s2 and so forth and adding up, we get

λs2π(s) − (λ + µ)s(π(s) − π0) + µ(π(s) − π0) − λπ0s = 0,

hence

π(s)(λs2 − (λ + µ)s + µ) = π0µ(1 − s).

This implies

π(s) =π0µ(1 − s)

λs2 − (λ + µ)s + µ=

π0µ(1 − s)

(s − 1)(λs − µ)=

π0µ

µ − λs.

To find π0 recall that π(1) = 1 hence

π0 =µ − λ

µ= 1 − ρ,

and we infer that

π(s) =(1 − ρ)µ

µ − λs=

1 − ρ

1 − ρs.

To recover the πn we expand π(s) as a power series and equate the coefficients,

π(s) = (1 − ρ)∞∑

n=0

ρnsn,

hence πn = (1 − ρ)ρn. The mean queue length in the long term can be found using

π′(s) =∞∑

n=1

nsn−1πn, and π′(s) =ρ(1 − ρ)

(1 − ρs)2.

which implies that the mean queue length is

∞∑

n=1

nπn = π′(1) =ρ

1 − ρ.

16

Example: The pure birth process.

Recall that the pure birth process X is a continuous time Markov chain with Q-matrix

Q =

−λ λ 0 0 . . .0 −2λ 2λ 0 . . .0 0 −3λ 3λ 0 . . .

0 . . . 0. . .

. . .. . .

.

This process is strictly increasing, and hence transient, we show that, if X(0) = 1, then

IP{X(t) = n} = e−λt(1 − e−λt)n−1 for n = 1, 2, 3, . . . , (2.4.1)

i.e. X(t) is geometrically distributed with parameter e−λt. Note that this shows that, as t → ∞,

IP{X(t) ≤ n} =n∑

k=1

e−λt(1 − e−λt)k−1 = 1 − (1 − e−λt)n −→ 0,

hence the process X(t) converges to infinity.

To prove (2.4.1) we use the resolvent method. We now write ρ for the Laplace parameter, hence wehave to invert

ρI − Q =

ρ + λ −λ 0 0 . . .0 ρ + 2λ −2λ 0 . . .0 0 ρ + 3λ −3λ 0 . . .

0 . . . 0. . .

. . .. . .

.

The coefficients rin, n = 1, 2, . . . of the inverse matrix hence satisfy

r11(ρ + λ) = 1, and r1,n−1(−(n − 1)λ) + r1,n(ρ + nλ) = 0, for n ≥ 2.

Solving this gives

r1n =(n − 1)λ

ρ + nλr1,n−1 = λn−1(n − 1)!

n∏

k=1

1

ρ + kλ.

We use partial fractions, which means finding ak, k = 1, . . . , n such that

r1n =n∑

k=1

ak

ρ + kλ.

We thus get the equations

λn−1(n − 1)! =n∑

k=1

ak

∏

j 6=k

(ρ + jλ).

Plugging in ρ = −λk gives

λn−1(n − 1)! = ak

∏

j 6=k

(λ(j − k)),

hence

ak =(n − 1)!∏j 6=k(j − k)

=

(n − 1

k − 1

)(−1)k−1.

17

Now we have

r1n =n∑

k=1

(n − 1

k − 1

)(−1)k−1 1

ρ + kλ.

Inverting the Laplace transform yields

p1n(t) =n∑

k=1

(n − 1

k − 1

)(−1)k−1e−kλt = e−λt

n−1∑

k=0

(n − 1

k

)(−1)ke−kλt = e−λt(1 − e−λt)n−1,

by the binomial formula. This verifies (2.4.1).

Example: The Poisson process.

Recall that the Poisson process X is a continuous time Markov chain with I = {0, 1, 2, . . .} andQ-matrix

Q =

−λ λ 0 0 . . .0 −λ λ 0 . . .0 0 −λ λ 0 . . .

0 . . . 0. . .

. . .. . .

.

Again the process is strictly increasing, and hence transient, we show that, if X(0) = 0, then

IP{X(t) = n} = e−λt (λt)n

n!for n = 0, 1, 2, . . . , (2.4.2)

i.e. X(t) is Poisson distributed with parameter λt.

To prove (2.4.2) we again use the resolvent method. Writing ρ for the Laplace parameter, we have toinvert

ρI − Q =

ρ + λ −λ 0 0 . . .0 ρ + λ −λ 0 . . .0 0 ρ + λ −λ 0 . . .

0 . . . 0. . .

. . .. . .

.

The coefficients rin, n = 1, 2, . . . of the inverse matrix hence satisfy

r00(ρ + λ) = 1, and r0,n−1(−λ) + r0,n(ρ + λ) = 0, for n ≥ 1.

Solving this gives

r0n =λ

ρ + λr0,n−1 =

λn

(λ + ρ)n+1.

Instead of inverting the Laplace transform we just show that the Laplace transform of the right handside in (2.4.2) agrees with the above, so that the uniqueness theorem can be applied. Indeed,

p0,n(ρ) =

∫ ∞

0e−ρte−λt (λt)n

n!dt

=λn

n!

∫ ∞

0e−(λ+ρ)ttn dt =

λn

(λ + ρ)n+1,

by partial integration (n times). This verifies (2.4.2).

18