Class Notes

Stochastic Manufacturing and Service Systems

ISyE 3232 – Spring 2002

January 2002

c©Antonio Pacheco

1

This notes are intended to support the lectures of ISyE 3232 (Stochastic Manufacturing andService Systems), as a supplement to the current textbook (Modelling, Analysis, Design, andControl of Stochastic Systems, by V.G. Kulkarni) or a textbook aiming at similar goals.

You may study these notes either before or after reading the textbook. If the textbook is readfirst, then these notes will highlight the relevant results, present additional examples, and giveadditional insight on the models and systems studied and their properties. Please, beware that theorder of material in these notes and in the textbook is not always the same.

In the notes I will try to respect the following two principles for each relevant computationalobjective of the course:

(i) To give an easy (and efficient, most of the time) algorithm to achieve the envisaged compu-tational goal; and

(ii) To give one example of application of the proposed algorithm.

There will not be put a strong emphasis on giving several examples of the same kind since: addi-tional examples are usually provided in the textbook and the best way to master a computationalprocedure it to actually executing it. We can perform the procedure using other examples of mod-els and systems considered in the notes and the textbook, textbook problems, or (even better) byconstructing your own examples and/or questions.

As you know, you should be at easy with the material covered in the first four chapters of thetextbook, respectively: Probability, Univariate Random Variables, Multivariate Random Variables,and Conditional Probability and Expectations. It will also help you to review this material thereading of the excellent notes of Prof. Richard Serfozo for ISyE 2027 (Basic Probability Problems),which are available online at http://www.isye.gatech.edu/˜rserfozo/isye2027/2027BasicPP.pdf.

2

Chapter 1

Discrete Time Markov Chains

In this course we will study the evolution of stochastic systems (systems whose evolution involvessome type of randomness). Three examples of stochastic systems are: the Dow Jones stock index,the arrival of orders for new subscriptions of telephone lines; the production of manufactured itemsby a machine with random production times (and/or with random supply of parts).

As we know from ISyE 2027, to study stochastic systems it is usually sufficient to look at thesystem in a macroscopic way, recording numerical functions of the state of the system – whichare called random variables or random vectors. As in this course we will be interested in theevolution of stochastic systems along time we may need to be able to describe the state of thesystem at an arbitrary number of chosen time instants. If the observed characteristic of the systemat time t is a random variable Xt, the state of the system at an arbitrary number of chosen timeinstants t1, t2, t3, . . . , is a random sequence Xt1 , Xt2 , Xt3 , . . .. Thus, we must be able to describethe distribution of random sequences. In general this task may be extremely complex. However, itis fairly simple for the type of models we will first study: Discrete Time Markov Chains (DTMCs).

We next give some motivation for the consideration of DTMCs, starting with modelling issuesand finishing with some justification of the broad application of these models.

1.1 Motivation

Recall that a random variable is a real-valued function of the result of a random experiment. If,e.g., the random experiment consists in observing the arrival of orders to a make-to-order system,then possible random variables we may be interest in studying are: the number of orders that arriveduring a particular day, the instant of arrival of the first order; and the maximum number of dailyorder arrivals during the first semester. Note that the first and third of these random variables arediscrete random variables, whereas the second random variable is a continuous random variable.

A random sequence is a sequence of random variables. The probability distribution of a randomsequence X = {Xn, n ∈ IN0} is characterized by the probability distributions of the random vectors(X0, X1, . . . , Xr) for nonnegative integer r. Thus, in general the stochastic modelling of a randomsequence (i.e., characterization of its probability distribution) may be a very difficult task. Wewill use the make-to-order example to motivate how this can be avoided in practice for some veryimportant systems.

Suppose that in the make-to-order system: the orders arrive all at the beginning of each day,in each day the system produces a number of items equal to the minimum of 2 (the production

3

capacity) and the number of back-ordered items (number of ordered items that have not yet beendelivered). Let Yi denote the number of item orders that arrive at the beginning of the ith day,i = 0, 1, 2, . . .. Similarly, let Xi denote the number of back-ordered at the end of the ith day,i = 0, 1, 2, . . ., with the items produced during the day having been delivered. The random sequenceY = {Yn, n ∈ IN0} is an input of the make-to-order system, whereas the random sequence X ={Xn, n ∈ IN0} provides the most important performance behavior of the system.

Let us suppose that the number of orders that arrive in different days are independent andidentically distributed, so that Y is a sequence of independent and identically distributed randomvariables; you may think on conditions that would make this assumption plausible. Then, if we let{pn, n ∈ IN0} denote the probability mass function of the number of orders that arrive in a day,the probability mass function of (Y0, Y1, . . . , Yr) is

p(Y0,Y1,...,Yr)(y0, y1, . . . , yr) =r∏

i=0

pyi (1.1)

for any r, y0, y1, . . . , yr ∈ IN0. This shows that there is no problem whatsoever in characterizing theprobability distribution of sequences of independent random variables, as opposed to what happens,in general, with sequences of dependent random variables.

As concerns the random sequence X = {Xn, n ∈ IN0}, it is simple to conclude that it is nota sequence of independent random variables since, e.g., if Xn = 3 (i.e., there are 3 back-ordereditems at the end of the nth day) then Xn+1 ≥ 1 (i.e., there will be necessarily back-ordered itemsat the end of the (n + 1)th day) as the daily item production capacity is 2. However, since if weknow the number of back-ordered items at the end nth day, then the number of back-ordered itemsat the end of days 0, 1, 2, . . . , n − 1 will have no effect on the number of back-ordered items therewill be at the end of day (n + 1), the joint probability mass function of (X0, X1, . . . , Xr) is

p(X0,X1,...,Xr)(x0, x1, . . . , xr) = pX0(x0)r∏

i=1

pXi|Xi−1=xi−1(xi) (1.2)

for x0, x1, . . . , xr ∈ IN0. The statement made about the random sequence X is equivalent to saythat X has the Markov property; i.e.,

· The distribution of Xn+1 given (X0, X1, . . . , Xn) is the same as the distribution of Xn+1 given(only) Xn, for any n ∈ IN0.

We may conclude that in fact the number of back-ordered items at the end of day (n + 1) isonly a function of the number of back-ordered items at the end of day n and the number of ordersthat arrive in day (n+1), namely:

Xn+1 = max(0, Xn + Yn+1 − 2). (1.3)

Moreover, the random sequence X possesses the homogeneity property:

· The distribution of Xn+1 given Xn is the same for all n ∈ IN0.

This follows in view of (1.3), since Y0, Y1, Y2, . . . are independent and identically distributed randomvariables. More precisely,

P(Xn+1 = j|Xn = i) =

pi+j−2 j ≥ max(1, i− 2)∑k−il=0 pl i ≤ 2 ∧ j = 0

0 otherwise

(1.4)

4

for all n ∈ IN0, where as defined before {pn, n ∈ IN0} is the probability mass function of the randomvariables Y0, Y1, . . ..

Since the random sequence X = {Xn, n ∈ IN0} satisfies the Markov and homogeneity propertiesand the random variables of the sequence take nonnegative integer values (and so the state spaceof X is discrete), X is called an (homogeneous) discrete time Markov chain (DTMC). The DTMCX has state space IN0 and transition probability matrix P = [pij ], where

pij = P(Xn+1 = j|Xn = i), i, j ∈ IN0

regardless of the value of the nonnegative integer n. Moreover,

P =

p00 p01 p02 p03 p04 . . .p10 p11 p22 p13 p14 . . .p20 p21 p22 p13 p24 . . .p30 p31 p32 p23 p34 . . .p40 p41 p42 p43 p44 . . .. . . . . . . . . . . . . . . . . .

=

p0 + p1 + p2 p3 p4 p5 p6 . . .p0 + p1 p2 p3 p4 p5 . . .

p0 p1 p2 p3 p4 . . .0 p0 p1 p2 p3 . . .0 0 p0 p1 p2 . . .

. . . . . . . . . . . . . . . . . .

. (1.5)

Note that, in view of (1.2) and (1.4), it follows that the joint probability mass function of(X0, X1, . . . , Xr) is

p(X0,X1,...,Xr)(j0, j1, . . . , jr) = aj0

r∏

i=1

pji−1ji (1.6)

where aj = P(X0 = j), for j ∈ IN0. Thus, the probability distribution of the random sequence X ischaracterized (completely) by the vector of initial probabilities a = [a0 a1 a2 . . .] and the transitionprobability matrix P .

We may therefore conclude that the Markov and homogeneity property provide a “friendly” wayto characterize the probability distribution of random sequences as Markov chains. Moreover, wewill see along the course that Markov chains may be used to model successfully several stochasticsystems of interest in practice, like the make-to-order example considered. We will consider onlyhomogeneous Markov chains and most of the time these will have finite state space.

1.2 Modelling of DTMCs

In this section we address the problem of identifying random sequences which may be characterizedas a DTMC and model them as such, i.e., to determine its state space and its transition probabilitymatrix, and sometimes also its initial probabilities (state probabilities at time zero).

Problem 1 Given a random sequence be able to decide if it is a DTMC and, if so, determine itsstate space, its transition probability matrix, and its initial probability vector. ¥

We first note that the random sequence X = {Xn, n ∈ IN0} described in the previous sectionis a DTMC with state space IN0 and transition probability matrix as given in (1.5), and thatY = {Yn, n ∈ IN0} is also a DTMC with state space IN0 (if the random variables Y0, Y2, . . . , takevalues on IN0) and transition probability matrix

p0 p1 p2 . . .p0 p1 p2 . . .p0 p1 p2 . . .. . . . . . . . . . . .

(1.7)

5

so that the transition probability matrix of Y has identical rows.The next example illustrates the situation in which the Markovian nature of the random se-

quence is given as an a priori fact.

Example 1 A small shop as a unique cash register machine. The owner of the shop checks the cashregister every day before opening the shop. From past experience, he knows that the cash registeris operational 99% of times he does the check. Whenever the cash register is not operational, hecalls the company that gives assistance to the cash register to take it for repair. Each day the cashregister is with this company, there is, regardless of the time the cash register has been repairing, a90% probability that the cash register is returned to the shop at the end of the day. As it should,the cash register is initially operational. Decide if the state (operational or not operational) of thecash register at the beginning of each day may be described using a DTMC. If so, obtain its statespace, initial probability vector and transition probability matrix. ¥

Solution: From the statements, it follows that the (operational) state of the cash register at thebeginning of a given day depends on its states at the previous days only through its state at theprevious day. Namely, the probability that the cash register is operational is 0.99 (0.90) if the cashregister was operational (not operational) at the beginning of the previous day. Thus if we let

Xn =

{1 the cash register is operational at the beginning of day n

0 otherwise

for n ∈ IN0, then X = {Xn, n ∈ IN0} is a DTMC with state space {0, 1}, initial probability vectora = [0 1] since the machine is initially operational, and transition probability matrix

P =[0.10 0.900.01 0.99

]

since, as written before in words, P(Xn+1 = 1|Xn = 1) = 0.99 and P(Xn+1 = 1|Xn = 0) = 0.90,for any n ∈ IN0. ¥

The examples that we have considered so far illustrate the two most common ways to identifyas DTMCs random sequences taking values in a discrete set:

· Based on a priori considerations. This is holds in situations where from either log-ical/technical reasons of from past experience its is known that the random sequence ofinterest is Markovian (as, e.g., in Example 1).

· Based on the recursion approach. The random sequence X = {Xn, n ∈ IN0} is a DTMCif we can establish that for some deterministic function g and some random (vector) sequenceY = {Yn, n ∈ IN0}

Xn+1 = g(Xn,Yn+1), n ∈ IN0 (1.8)

where, given Xn, Yn+1 is independent of (X0, X1, . . . , Xn−1) (this holds in particular if Yn+1

is independent of (X0, X1, . . . , Xn), which is a common situation in applications).

Example 2 A particular example of the use of the recursion approach was given in the previoussection for the make-to-order system. There, Xn is the number of back-ordered items at the endof day n and Yn is the number of items ordered in day n. The recursion associated to the sequenceof numbers of back-ordered items at the beginning of each day is (1.3), so that in this exampleg(x, y) = max(0, x + y − 2), for x, y ∈ IN0, and Yn+1 is independent of (X0, X1, . . . , Xn). ¥

6

Exercise 1 For each of the examples of DTMCs given in Section 5.3 of the textbook, identifywhich of the two presented ways to identify DTMCs is used, and specify the function g for theexamples where the recursion approach is used. ¥

1.3 Transient quantities

In this section we address the problem of computing transient quantities (i.e., quantities that dependon the joint distribution at a finite set of time instants) associated with a DTMC. Before statingthe problem and giving examples, we introduce some notation along with the main results that areuseful to characterize transient distributions of a DTMC.

Given a DTMC X = {Xn, n ∈ IN0} with state space S, initial probability vector a and transitionprobability matrix P , we let the vector a(n) = [a(n)

i ]i∈S denote the probability vector of the state ofthe DTMC at time n, Xn, and the matrix P (n) = [pij ]i,j∈S denote the n-step transition probabilitymatrix of the DTMC, so that

a(n)i = P(Xn = i), p

(n)ij = P(Xm+n = j|Xm = i)

for i, j ∈ S and m,n ∈ IN0. The fact that X is a DTMC leads, using the total probability law, tothe Chapman-Kolmogorov equations

P (n+m) = P (n) P (m), n, m ∈ IN0

which, in view of P (0) = I, lead to:

P (n) = Pn and a(n) = aPn

for all n ∈ IN0. As a consequence, given r ∈ IN and 0 = n0 ≤ n1 < n2 < . . . < nr the jointprobability mass function of (Xi1 , Xi2 , . . . , Xir) is given by

P(Xn1 = i1, Xn2 = i2, . . . , Xnr = ir) =∑

i0∈S

ai0

r∏

j=1

[Pnj−nj−1 ]ij−1ij (1.9)

= a(n1)i1

r∏

j=2

[Pnj−nj−1 ]ij−1ij (1.10)

for i1, i2, . . . , ir ∈ S. This joint probability mass function may then be used to derive any quantitiesthat are functions of the distribution of (Xi1 , Xi2 , . . . , Xir) (like probabilities, expected values,variances, covariances and correlations).

Problem 2 Given the initial probability vector and the transition probability matrix of a DTMC,compute quantities associated to the state of the DTMC at a finite set of instants.

Example 3 Consider the DTMC X described in the solution of Example 1. Compute the proba-bilities of the following events:

· A = “the cash register is operational on day 4”; and

· B = “during the first week (days 0, 1, . . . , 6) the cash register is operational at least 6 days”.

7

Compute also E[|X3−X2|] and the expected number of days the cash register is operational duringdays 2 and 3. ¥

Solution: First note that

P =[0.10 0.900.01 0.99

], P 2 =

[0.0190 0.98100.0109 0.9891

], P 4 =

[0.01105390 0.988946100.01098829 0.98901171

].

Now, since the initial probability vector is a = [0 1], we have

P(A) = P(X4 = 1) = P(X4 = 1|X0 = 1) = p(4)11 = 0.98901171.

Similarly, it follows that P(B) = P(∑6

i=0 Xi ≥ 6) is equal to

P(B) =P(Xi = 1, i = 0, 1, . . . , 6) +6∑

i=0

P(Xi = 0, Xj = 1, j ∈ {0, 1, . . . , 6} \ {i})

=a1 p611 + a0 p01 p5

11 + 5× (a1 p10 p01 p411) + a1 p5

11 p10

=1× 0.996 + 0× 0.90× 0.995 + 5× (1× 0.01× 0.90× 0.994) + 1× 0.995 × 0.01≈ 0.99421687.

Finally, we want to compute E[|X3−X2|] and E[X2 + X3]. Since a(2) = [0.0109 0.9891] (why?)and, for i, j = 0, 1, P(X2 = i,X3 = j) = a

(2)i pij we have

[P(X2 = i,X3 = j)] =[0.0109× 0.10 0.0109× 0.900.9891× 0.01 0.9891× 0.99

]=

[0.001090 0.0098100.009891 0.979209

].

Thus,

E[|X3 −X2|] = P(X2 6= X3) = P(“The cash register machine is in different states in days 2 and 3”)

=1∑

i=0

1∑

j=0

|j − i|P(X2 = i, X3 = j)

= P(X2 = 0, X3 = 1) + P(X2 = 1, X3 = 0)= 0.009810 + 0.009891 = 0.019701.

and,

E[X2 + X3] =1∑

i=0

1∑

j=0

(i + j)P(X2 = i,X3 = j)

= P(X2 = 0, X3 = 1) + P(X2 = 1, X3 = 0) + 2×P(X2 = 1, X3 = 1)= 0.009810 + 0.009891 + 2× 0.979209 = 1.978119.

Observe that the computations made show that precision may be lost when computing successivepowers of a transition probability matrix. In practice, this may be partially avoided by normalizingthe rows of the power matrices so that the rows of their resulting approximations have unit sums,thus guaranteeing that these approximations are stochastic matrices, as the powers themselves. ¥

At this point, and as motivation for what comes ahead, we may want to read carefully the casestudy “The Cash Register Machine Contract” presented at the end of the chapter and discussed inclass.

8

1.4 Expected values

In general, expected values of functions of the state of a DTMC X at a finite number of instants,n1, n2, . . . , nk, may be computed by using the probability mass function of the random vector(Xn1 , Xn2 , . . . , Xnk

) using (1.9)-(1.10), as shown in the previous section. However, a usually sim-pler approach can be used when the functions of interest correspond to (eventually accumulated)cost/rewards incurred/collected when the DTMC makes certain transitions or visits certain states.

As an example, in Example 3 we could have computed the expected number of days the cashregister is operational during days 2 and 3, E[X2 + X3], by noting that the expected value is alinear operator, which implies that E[X2 + X3] = E[X2] + E[X3], and noting that E[X2] (E[X3])is the expected value associated to the probability row vector a(2) (a(3)). Using these facts, analternative way to compute E[X2 + X3] is

E[X2 + X3] = E[X2] + E[X3]

=[a

(2)0 a

(2)1

] [01

]+

[a

(3)0 a

(3)1

] [01

]

=[a0 a1

](P 2 + P 3)

[01

](1.11)

=[0 1

]([0.0190 0.98100.0109 0.9891

]+

[0.011710 0.9882900.010981 0.989019

])[01

]

= 1.978119.

The formula (1.11) can be generalized to any finite number of instants. If X is a DTMC thatassumes real values j0, j1, . . . , jn and has initial probability row vector a and transition probabilitymatrix P , and T is a finite set of nonnegative integers, then

E

[∑

k∈T

Xk

]=

∑

k∈T

E[Xk] =[a0 a1 . . . an

](∑

k∈T

P k

)

j0

j1

. . .jn

(1.12)

where ar = P(X0 = jr), for r = 0, 1, . . . , n. A common situation is when T = {0, 1, 2, . . . , m} forsome positive integer m.

Problem 3 Given the initial probability vector and the transition probability matrix of a DTMC,compute expected values (or derived quantities) associated to the cost/rewards incurred/collectedwhen the DTMC makes certain transitions or visits certain states at a finite set of instants. ¥

Example 4 Consider the DTMC X described in the solution of Example 1 and used in Example 3.Compute the expected number and fraction of days the cash register is operational during days0, 1, 2, . . . , n for n equal to 10, 30 (one month), 364 (one year), and 4382 (twelve years).Solution: Let en (fn) denote the expected number (fraction) of days the cash register is operationalduring days 0, 1, 2, . . . , n. In view of (1.12) and since the cash register is initially operational (i.e.,X has initial probability row vector a = [0 1]), we have

en = E

[n∑

k=0

Xk

]= a

n∑

k=0

P k

[01

]=

[0 1

](

n∑

k=0

[0.10 0.900.01 0.99

]k)[

01

]

9

and fn = en/(n + 1); note that en =∑n

k=0 p(k)11 . Thus, it follows that

E

[10∑

k=0

Xk

]=

[0 1

] [1.2077 9.79230.1088 10.8912

] [01

]= 10.8912

E

[30∑

k=0

Xk

]=

[0 1

] [1.42748460 29.572515400.32858350 30.67141650

] [01

]= 30.67141650

E

[364∑

k=0

Xk

]=

[0 1

] [5.09781427 359.902185733.99891317 361.00108683

] [01

]= 361.00108683

E

[4382∑

k=0

Xk

]=

[0 1

] [49.2516604 4333.7483395748.1527593 4334.84724067

] [01

]= 4334.84724067

and

111

E

[10∑

k=0

Xk

]=

[0 1

] [0.10979131 0.890208690.00989121 0.99010879

] [01

]= 0.99010879

131

E

[30∑

k=0

Xk

]=

[0 1

] [0.04604789 0.953952110.01059947 0.98940053

] [01

]= 0.98940053

1365

E

[364∑

k=0

Xk

]=

[0 1

] [0.01396661 0.986033390.01095593 0.98904407

] [01

]= 0.98904407

14383

E

[4382∑

k=0

Xk

]=

[0 1

] [0.01123697 0.988763030.01098626 0.98901374

] [01

]= 0.98901374

Note that the sequence of fraction of days the cash register is operational during days 0, 1, 2, . . . , nseems to be converging as n tends to infinity. This is indeed true; in fact

Pn ≈[0.01098901 0.989010990.01098901 0.98901099

]

for n ≥ 10. The value for which the fraction of days the cash register is operational during days0, 1, 2, . . . , n is converging is 0.90/(0.01+0.90) ≈ 0.98901099. This fact could not have been guessedfrom the computation of the values of the sequence 1

n+1E [∑n

k=0 Xk]. ¥

Let X denote a DTMC with state space {j0, j1, . . . , jn}, initial probability row vector a and tran-sition probability matrix P , and C be a finite set of nonnegative integers. If instead of E

[∑k∈C Xk

],

as in (1.12), we want to compute the expected cost/reward collected over the finite set of instants T ,where a cost/reward cr = c(jr) is collected at each visit to state jr, then the expected accumulatedcost/reward over the set of instants C is E

[∑k∈C c(Xk)

]and

E

[∑

k∈T

c(Xk)

]=

∑

k∈T

E[c(Xk)] =[a0 a1 . . . an

](∑

k∈T

P k

)

c0

c1

. . .cn

. (1.13)

10

If there is a discount α-factor per time unit for the costs/rewards, so that the accumulatedcost/reward over the set of instants T would be

∑k∈T αk c(Xk), we will have instead

E

[∑

k∈T

αk c(Xk)

]=

∑

k∈T

E[αk c(Xk)] =[a0 a1 . . . an

](∑

k∈T

[αP ]k)

c0

c1

. . .cn

. (1.14)

Example 5 Consider the DTMC X of Example 4. Suppose the owner of the cash register buysit by 2000 units of a currency but makes a contract with the seller such that the seller has toreimburse him 10 units for each day the cash register is being repaired (i.e., it is not operational)during the first 12 years (days 0, 1, . . . , 4382). Compute the expected real cost of the cash register(at current prices) assuming that the currency suffers from hyperinflation of 70% a year.Solution: Since the inflation rate is 70% a year and the solution of (1 + x)365 = 1.70 is x =0.00145483, the daily percentage rate of inflation is 0.145483 and so the current value of one unitof the currency in k days from now into the future is 1/1.00145483k. Thus the expected real cost ofthe cash register, EC, is (with the notation of Example 4) 2000−E[

∑4382k=0 1.00145483−k 10(1−Xk)];

i.e., 2000 minus the expression (1.14) with α = 1/1.00145483, c0 = c(0) = 10, and c1 = c(1) = 0.Thus

EC = 2000− [0 1

][

4382∑

k=0

(1

1.00145483

[0.10 0.900.01 0.99

])k][

100

]

= 2000− [0 1

] [8.638 678.5517.539 679.650

] [100

]

= 2000− 75.39 ≈ 1924.61.

Thus we conclude that at current prices, the owner should expect to be reimbursed in only 75.39currency units, and thus the expected real cost of the cash register is 1924.61. ¥

Remark 1 We should note that the matrix E(C) =∑

k∈CT P k that appears in (1.13) is the matrixof expected number of visits to each state of DTMC over the set of instants T for each initial state.Thus, e.g., in Example 4,

E({0, 1, 2, . . . , 4382}) =[49.2516604 4333.7483395748.1527593 4334.84724067

].

In particular, if the cash register is initially not operational it is expected that it will not beoperational approximately 49.25 days during the first 12 years. ¥

Note that if a cost/reward crs = c(jr, js) is collected at each transition from state jr tostate js, then the expected accumulated cost/reward over the finite set of instants T ⊆ IN isE

[∑k∈CT c(Xk−1, Xk)

]and

E

[∑

k∈T

c(Xk−1, Xk)

]=

∑

k∈T

E[c(Xk−1, Xk)] =[a0 a1 . . . an

](∑

k∈T

P k−1

)[P • C]1 (1.15)

where P • C denotes the Schur or entrywise multiplication of matrices; i.e., [P • C]rs = prs crs =P(Xk = js|Xk−1 = jr) c(jr, js), and 1 is a column vector of ones. If there is a discount α-factor

11

per time unit for the costs/rewards, so that the accumulated cost/reward over the set of instantsT would be

∑k∈T αk c(Xk−1, Xk), we will have instead

E

[∑

k∈T

αk c(Xk−1, Xk)

]=

∑

k∈T

E[αk c(Xk−1, Xk)]

=[a0 a1 . . . an

](∑

k∈T

[αP ]k−1

)[(αP ) • C]1. (1.16)

Note that if T is a finite set of positive integers, then (1.13)-(1.14) are particular instances of(1.15)-(1.16) with the cost/reward matrix C having all rows equal, namely, crs = c(jr, js) = cs, forr, s = 0, 1, 2, . . . , n.

Observe in addition that (1.13)-(1.16) remain valid if deterministic cost/rewards are replacedin matrix C by expected values of cost/rewards associated with visits to states of the DTMC ortransitions between states in case some of the cost/rewards are random (but their distribution doesnot change with time).

Example 6 A telecommunications switch has a buffer which can store a maximum of 10 packets.In each time slot a binomial number of packets with parameters 5 and 0.2 arrives to the switchand the number of packets arriving in distinct time slots are independent. Packets that find nospace free in the buffer when arriving to the switch are lost. At the beginning of each time slot apacket is removed from the buffer and processed, if there are packets in the buffer. This examplewas described in Computational Problem 5.5.

Compute the expected number of packets lost in the first 1000 time slots if the buffer is initiallyfull.Solution: Let Xn+1 denote the number of packets in the buffer at the end of time slot n and An+1

the number of packets that arrive during time slot n, for n = 0, 1, 2, . . .. {An, n = 1, 2, . . .} is asequence of independent and identically distributed random variables with Binomial distributionwith parameters (5, 0.2), so that

ak = P(An = k) =

0.85 k = 00.84 k = 10.4× 0.83 k = 20.08× 0.82 k = 34× 0.24 k = 40.25 k = 50 otherwise

.

Moreover, as Xn+1 = min(max(0, Xn−1)+An+1, 10) and An+1 is independent of Xn, we conclude,by proceeding as in Conceptual Problem 5.10, that {Xn, n ≥ 0} is a DTMC with state space{0, 1, 2, . . . , 10} and transition probability matrix P as given below.

The number of packets lost at the nth time slot is

Ln+1 = max(0, max(0, Xn − 1) + An+1 − 10) =

{0 Xn ≤ 6 ∨Xn+1 < 10Xn + An+1 − 11 Xn ≥ 7 ∧Xn+1 = 10

.

12

P =

a0 a1 a2 a3 a4 a5 0 0 0 0 0a0 a1 a2 a3 a4 a5 0 0 0 0 00 a0 a1 a2 a3 a4 a5 0 0 0 00 0 a0 a1 a2 a3 a4 a5 0 0 00 0 0 a0 a1 a2 a3 a4 a5 0 00 0 0 0 a0 a1 a2 a3 a4 a5 00 0 0 0 0 a0 a1 a2 a3 a4 a5

0 0 0 0 0 0 a0 a1 a2 a3∑5

i=4 ai

0 0 0 0 0 0 0 a0 a1 a2∑5

i=3 ai

0 0 0 0 0 0 0 0 a0 a1∑5

i=2 ai

0 0 0 0 0 0 0 0 0 a0∑5

i=1 ai

For j, k = 0, 1, 2, . . . , 10, we let cjk denote the expected number of packets lost in a time slot inwhich the number of packets in the buffer transits from j to k, so that

cjk = E[Ln+1|Xn = j, Xn+1 = k] =

{0 j ≤ 6 ∨ k < 10E[An+1 − (11− j)|An+1 ≥ 11− j] j ≥ 7 ∧ k = 10

.

Thus the matrix of the expected number of packets lost by transition of X is

C =

0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0.0476190 0 0 0 0 0 0 0 0 0 0.1215470 0 0 0 0 0 0 0 0 0 0.2472590 0 0 0 0 0 0 0 0 0 0.487387

.

Since the buffer is initially full, the expected number of packets lost in the first 1000 time slots is

E

[1000∑

k=1

Lk

]=

[0 0 0 0 0 0 0 0 0 0 1

](

1000∑

k=1

P k−1

)[P • C]1 = 42.0399. ¥

1.5 Irreducibility and aperiodicity

In this section we introduce accessory concepts that will be very useful to derive long-run andlimit quantities associated to a DTMC, like long-run proportion of time spent in each state, thelimit probability of being in a state, and the long run cost per unit time of some (cost) functionassociated to the states visited by the DTMC.

Given a DTMC with state space S and transition probability matrix P , we may be interestedin knowing:

13

• If it is possible to visit state j starting from state i; i.e. state j is accessible from state i:

j is accessible from i ⇔ ∃n ≥ 0 : p(n)ij > 0.

If for any i, j ∈ S, j is accessible from state i, then X is said to be irreducible.

• At which instants it is possible to return to state i; i.e. the set

R(i) = {n ≥ 1 : p(n)ii > 0}.

The period of state i, d(i), is then the greatest common denominator of the set R(i) and d(i)is made equal to 1 if R(i) is empty. If the period of i is 1, then i is said to be aperiodic. Ifall states of X are aperiodic than X is said to be an aperiodic DTMC.

The study of questions of the type of the two stated plays an important role in what is generallycalled state classification of DTMCs. This study may be quite involved for DTMCs with large-finiteor infinite state spaces, although it may be simple in particular cases. E.g., if the transition proba-bility matrix of a DTMC has only positive entries, then the DTMC is irreducible and aperiodic. Asanother example, the DTMC of the make-to-order system with transition probability matrix (1.5)

P =

p0 + p1 + p2 p3 p4 p5 p6 . . .p0 + p1 p2 p3 p4 p5 . . .

p0 p1 p2 p3 p4 . . .0 p0 p1 p2 p3 . . .0 0 p0 p1 p2 . . .

. . . . . . . . . . . . . . . . . .

is irreducible if pi > 0 for all i ∈ IN0, i.e., there may arrive any number of orders during a day. Thisfollows since (then) for i ∈ IN0, pij > 0 for j ≥ max(0, i− 2) and

p(i−j)ij ≥ P(X1 = i− 1, X2 = i− 2, . . . , Xj−i = j|X0 = i) =

i∏

k=j+1

pk k−1 ≥ pj−i1 > 0

for 0 ≤ j ≤ i− 2. Moreover, as pii ≥ p2 > 0, the DTMC is aperiodic.In the course we are mainly interested in finite state DTMCs and, by reasons that will be

evident soon, on asserting that a DTMC is irreducible and aperiodic. Thus we will be interestedin solving the following generic problem.

Problem 4 Given the transition probability matrix of a (usually finite) DTMC, decide if the DTMCis irreducible and, if yes, if it is aperiodic. ¥

The following important facts make the solution of the previous problem simpler:

• A finite DTMC X with n + 1 states and transition probability matrix P is irreducible if andonly if

∑nk=0 P k has all entries positive.

• If X is irreducible, then all states of X have the same periodicity.

14

A particular case of a transition probability matrix with all entries positive is the one of Exam-ple 5,

P =[0.10 0.900.01 0.99

].

Thus, the DTMC X of Example 5 is irreducible and aperiodic.We also note that the condition

∑nk=0 P k having all entries positive is satisfied only and only

if∑n

k=0[A(P )]k has all entries positive, where A(P ) is the incidence matrix of the directed graphwith nodes corresponding to the states of the DTMC and having a (directed) arc going from nodei to node j if a one-step transition from state i to state j is possible, i.e., pij > 0. This method,which we will apply in Example 10, leads to simpler computations as it can be done using onlyinteger arithmetic.

Observe that the condition∑n

k=0 P k (∑n

k=0[A(P )]k) having all entries positive should be usedwith care when the state space of the DTMC is large, as in this case it is computationally demanding.

Example 7 A DTMC X with state space {1, 2, 3} and transition probability matrix P as givennext is irreducible since

∑2k=0 P k has all entries positive.

P =

0 .4 .61 0 01 0 0

, P 2 =

1 0 00 .4 .60 .4 .6

,

2∑

k=0

P k =

2 .4 .61 1.4 .61 .4 1.6

.

To compute the period of the DTMC let us look at returns to state 1. Since p11 = 0 and p(2)11 = 1

we conclude that it is impossible to return to state 1 after one step and it is sure to return to state1 after two steps. Using this reasoning recursively (and since it is sure to return to state 1 aftertwo steps) we conclude (show it!) that

p(n)11 =

{0 n odd1 n even

so that R(1) = {2, 4, 6, . . .} and the period of state 1 is two. Thus, X is an irreducible DTMC withperiod two. ¥

Example 8 A DTMC X with state space {1, 2, 3} and transition probability matrix P as givennext is reducible since

∑2k=0 P k has null entries: the entries (2, 1) and (3, 1).

P =

.7 .2 .10 .6 .40 .4 .6

, P 2 =

.49 .30 .210 .52 .480 .48 .52

,

2∑

k=0

P k =

2.19 .50 .310 2.12 .880 .88 2.12

.

Conversely since all diagonal entries of P are positive the DTMC X is aperiodic. ¥

When a finite state DTMC is irreducible and aperiodic we say that the DTMC is ergodic.

1.6 Long-run and limit related quantities

In this section we will derive long-run and limit quantities associated to a DTMC, like long-runproportion of time spent in each state, the limit probability of being in a state, and the long runcost per unit time of some (cost) function associated to the states visited by the DTMC. We startby presenting an important auxiliary result.

15

Lemma 1 (Stationary distribution of an irreducible DTMC)

If X is an irreducible DTMC with state space S and transition probability matrix P , then thereis at most one probability row vector that is solution of the system of linear equations

π = π P ; (1.17)

i.e.,πj =

∑

i∈S

πi pij , j ∈ S (1.18)

and ∑

j∈S

πj = 1. (1.19)

Moreover, if the initial distribution of X is π, then X is a stationary DTMC, i.e., (Xn1 , Xn2 ,. . . , Xnr) has the same distribution as (Xn1+m, Xn2+m, . . . , Xnr+m), for all r,m ∈ IN and n1, n2, . . . ,nr ∈ IN0 – in particular, the distribution of Xn is the same for all values of n. In this case X issaid to be a stationary DTMC and π is called the stationary probability row vector (or stationarydistribution) of X. If the irreducible DTMC has finite state space then the stationary distributionexists (and is unique). ¥

The equations (1.18) are known as the balance or steady-state equations because they can berewritten as

πj(1− pjj) =∑

i6=j

πi pij , j ∈ S (1.20)

so that, under the probability distribution π the probability of leaving state j equals the probabilityof entering state j, for any state j. In general, we can test if a probability row vector is a stationaryprobability row vector by checking if it satisfies (1.17) or (1.18). This holds also for countablestate DTMCs, and if the DTMC is irreducible there is the additional guarantee that a solution of(1.18)-(1.19) (if it exists) is unique.

Example 9 The irreducible DTMC X of Example 5, which has state space {0, 1}, has stationaryprobability row vector π = [π0 π1] = [1− π1 π1] such that

[1− π1 π1] = [1− π1 π1][0.10 0.900.01 0.99

]⇔ π1 = 0.90(1− π1) + 0.99π1

⇔ π1 =0.90

1 + 0.90− 0.99⇔ [1− π1 π1] = [0.01098901 0.98901099] . ¥

Note that in the previous example we substituted π0 by 1 − π1. In general, we can substituteone of the probabilities πj by 1 −∑

i6=j πi; by doing that, the number of equations of the systemπ = π P is the number of variables plus one, so that we may use the extra equation to perform anempirical check on the solution obtained. We give now another example of the computation of thestationary distribution of a DTMC.

Example 10 Items arrive at a machine shop to be processed in a deterministic fashion at therate of one per minute. Each item is, independently of others, non-defective with probability p.

16

Defective items are discarded and non-defective items are put in a bin waiting to be processed. Themachine processes two items simultaneously and takes two minutes to process them.

To describe the system we let Z = {Zn = (Xn, Yn), n ≥ 0}, where

Xn =

0 the machine is not processing items at the beginning of minute n

1 the machine started processing (two) items at the beginning of minute n

2 the machine started processing (two) items at the beginning of minute (n− 1)

and Yn the number of items that remain in the bin at the beginning of minute n. From theconditions stated, we may conclude that Zn can only be equal to (0, 0), (0, 1), (1, 0), (2, 0), and(2, 1). Moreover, if we let

Wn =

{0 the item that arrives at the beginning of minute n is defective1 the item that arrives at the beginning of minute n is non-defective

then Zn+1 = g(Zn, Wn+1) for n = 0, 1, . . ., where

g((0, 0), 0) = (0, 0) g((0, 0), 1) = (0, 1)g((0, 1), 0) = (0, 1) g((0, 1), 1) = (1, 0)g((1, 0), 0) = (2, 0) g((1, 0), 1) = (2, 1)g((2, 0), 0) = (0, 0) g((2, 0), 1) = (0, 1)g((2, 1), 0) = (0, 1) g((2, 1), 1) = (1, 0).

Since {Wn, n ≥ 0} is a sequence of independent and identically distributed random variableswith Bernoulli distribution with parameter p and Wn+1 is independent of Zn, for all n, {Zn =(Xn, Yn), n ≥ 0} is a DTMC with state space S = {(0, 0), (0, 1), (1, 0), (2, 0), (2, 1)}. Moreover,since P(Wn+1 = 1) = p and P(Wn+1 = 0) = 1−p, the transition probability matrix of {Zn, n ≥ 0}is

P =

1− p p 0 0 00 1− p p 0 00 0 0 1− p p

1− p p 0 0 00 1− p p 0 0

.

Here the transition probabilities were computed using the function g and the fact that

P(Zn+1 = (k, l)|Zn = (i, j)) =1∑

m=0

P(Wn+1 = m)×P(Zn+1 = (k, l)|Zn = (i, j),Wn+1 = m)

=1∑

m=0

P(Wn+1 = m)× 1{g((i,j),m)=(k,l)}

= P(Wn+1 = 0)× 1{g((i,j),0)=(k,l)} + P(Wn+1 = 1)× 1{g((i,j),1)=(k,l)}= (1− p) 1{g((i,j),0)=(k,l)} + p 1{g((i,j),1)=(k,l)}

for (i, j), (k, l) ∈ S.

17

Now, by constructing the incidence matrix A(P ) of the directed graph associated to the DTMCZ we obtain

A(P ) =

1 1 0 0 00 1 1 0 00 0 0 1 11 1 0 0 00 1 1 0 0

, [A(P )]3 =

1 3 2 1 11 3 2 1 11 3 2 1 11 3 2 1 11 3 2 1 1

.

Since [A(P )]3 has all entries positive we conclude that Z is irreducible. Moreover the probabilityrow vector

π =[π(0,0) π(0,1) π(1,0) π(2,0) π(2,1)

]=

[(1− p)2

21− p2

2p

2p(1− p)

2p2

2

]

is the (unique) stationary probability row vector of Z. Note that this solution could have beenobtained by guessing first π(1,0), followed by π(2,0) and π(2,1), then by π(0,0), and, finally, by π(0,1)

(justify why!). ¥

The next theorem states the main long-run and limit properties of irreducible and aperiodicDTMCs.

Theorem 1 Let X be an irreducible DTMC with state space S, transition probability matrix P ,and possessing (unique, and guaranteed to exist if X has finite state space) stationary distributionprobability row vector π = [πj , j ∈ S]. Then:

• The long-run and the expected long-run proportion of time X spends in state j are equal toπj; i.e.,

limn→+∞

∑nk=0 1{Xk=j}

n + 1= lim

n→+∞E[∑n

k=0 1{Xk=j}n + 1

]= πj (1.21)

for j ∈ S, regardless of the initial state, where 1A is the indicator function of A.

• The expected time elapsed between two successive visits to state j is 1/πj, for j ∈ S; i.e.,

E [min{n ≥ 1 : Xn = j}|X0 = j] =1πj

. (1.22)

• Suppose that at each unit of time the DTMC is in state j, independently of what happens atother instants, a reward with finite expected value cj is collected. Let Yk denote the rewardreceived at time unit k, so that cj = E[Yk|Xk = j].

Then, the long-run and expected long-run reward rate per time unit are equal to π c, wherec = [cj , j ∈ S] is the column vector of expected rewards, i.e.,

limn→+∞

∑nk=0 Yk

n + 1= lim

n→+∞E[∑n

k=0 Yk

n + 1

]=

∑

j∈S

πj cj = π c (1.23)

provided∑

j∈S |cj |πj < ∞.

• Suppose that at each transition of the DTMC between states i and j, independently of whathappens at other transitions, a reward with finite expected value cij is collected. Let Y(k−1,k)

denote the reward received at the transition that takes place from time k−1 to time k, so that

cij = E[Y(k−1,k)|Xk−1 = i,Xk = j].

18

Then, the long-run and expected long-run reward rate per transition are equal to π[P • C]1,where C = [cij ]i,j∈S is the matrix of expected rewards associated with transitions betweenstates of X and 1 is a column vector of ones, i.e.,

limn→+∞

∑nk=1 Y(k−1,k)

n= lim

n→+∞E[∑n

k=1 Y(k−1,k)

n

]=

∑

i∈S

∑

j∈S

πi pij cij = π [P • C]1. (1.24)

provided∑

i∈S

∑j∈S πi pij |cij | < ∞.

• If the DTMC X is aperiodic, then π is the limit distribution of X; i.e.,

limn→+∞P(Xn = j|X0 = i) = πj (1.25)

for all i, j ∈ S. ¥

Problem 5 Given an irreducible DTMC, compute its stationary distribution if it exists and, ifso, derive quantities associated with the long-run (expected long-run) occupation of states, long-run(expected long-run) reward rates associated with visits to states or state transitions and expectedtimes between successive visits to states. If, in addition, the DTMC is aperiodic compute its limitdistribution and associated measures. ¥

Example 11 For the irreducible DTMC X of Example 5, compute the long-run proportion oftime the DTMC stays in state 1 (i.e., the long-run proportion of days the cash register machine isoperational) by deriving the expected time to return to state 1 (i.e., the expected number of daysneeded to have the cash register operational after a day in which the cash register is operational).Solution: Recall that X has transition probability matrix

P =[0.10 0.900.01 0.99

].

For i = 0, 1, let Tij denote the first day (strictly after day 0) at which the DTMC visits state jstarting from state i,

Tij = [min{n ≥ 1 : Xn = j}|X0 = i]

so that T11 represents the number of days needed to have the cash register operational after a dayin which the cash register is operational.

The long-run proportion of days the cash register is operational is π1 = 1/E[T11]. We computeE[T11] by conditioning on X1, the state of the cash register at day 1:

E[T11] = E[E[T11|X1]] =1∑

i=0

P[X1 = i|X0 = 1]E[T11|X1 = i]

= p10 ×E[T11|X1 = 0] + p11 ×E[T11|X1 = 1].

Now, as T01 ∼ G(0.90),

E[T11|X1 = i] =

{1 + E[T01] i = 01 i = 1

=

{1 + 1

0.90 i = 01 i = 1

.

Thus,

E[T11] = 1 + p101

0.90= 1 +

0.010.90

=9190

and π1 = 90/91 ≈ 98.9%. ¥

19

Example 12 For the telecommunications switch of Example 6, compute the expected: long-runmean number of packets in the buffer, long-run number of packets lost per time slot, number oftime slots between successive time slots at which the buffer is full.Solution: The DTMC {Xn, n ≥ 0}, where Xn+1 denotes the number of packets in the bufferat the end of time slot n, characterized in Example 6 is irreducible and aperiodic. Thus, sinceit has finite state space S = {0, 1, 2, . . . , 10}, it has a unique stationary probability row vector,π = [π0 π1 . . . π10], solution of π = π P with

∑10k=0 πk = 1, where P is given in Example 6,

π = [0.0392157 0.0804611 0.0959518 0.0980839 0.0980541 0.09803730.0980392 0.0980392 0.0980392 0.0980392 0.0980392].

Thus, the expected long-run mean number of packets in the buffer is

π

012

. . .10

= 5.37059.

Moreover, the expected long-run number of packets lost per time slot is π[P • C]1, where C isgiven in Example 6, and

P • C =

0 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 0.000320 0 0 0 0 0 0 0 0 0 0.007040 0 0 0 0 0 0 0 0 0 0.064960 0 0 0 0 0 0 0 0 0 0.32768

.

As a result,

π[P • C]1 =[

0.0980392 0.0980392 0.0980392 0.0980392]

0.000320.007040.064960.32768

= 0.0392157.

Thus, we conclude that the expected long-run number of packets lost per time slot is 0.0392157,i.e., approximately 3.92%. But 0.0392157 is just the long-run proportion of time slots at which thebuffer is empty at the end of the time slot, π0. Explain why this result holds.

The expected number of time slots between successive time slots at which the buffer is full isequal to 1/π10 = 0.0980392−1 = 10.2. ¥

20

Example 13 Let Xn represent the state on an (1, 4) inventory system at the beginning of dayn, where the state is the number of items in stock. An (s, S) inventory system is such thatreplenishment orders are made only at the end of days at which the net inventory level dropsstrictly below the basestock level s; if the net inventory level at the end of a day if j < s, then areplenishment order of size S − j is made that pushes the inventory level to the restocking level Sat the beginning of the next day.

If we denote by An+1 the total number of items ordered in day n, then

Xn+1 =

{Xn −An+1 Xn −An+1 ≥ s

S Xn −An+1 < s

for n ∈ IN0. Thus, if the sequence of daily total number of items ordered, {Ak, k ≥ 1}, is asequence of independent and identically distributed random variables, independent of X0, thenX = {Xn, n ≥ 0} is a DTMC with state space S = {s, s + 1, . . . , S} and transition probabilities

pij = P(Xn+1 = j|Xn = i) =

P(A1 = i− j) s ≤ i < S ∧ s ≤ j < S∑l≥i−s+1 P(A1 = l) s ≤ i < S ∧ j = S

P(A1 = 0) +∑

l≥S−s+1 P(A1 = l) i = j = S

.

For the case where S = 4, s = 1, and

P(A1 = k) =

0.5 k = 00.3 k = 10.2 k = 2

compute the long-run mean inventory level, the long-run fraction of ordered items that are notdelivered in the day they are ordered, and the expected number of days between successive re-plenishment orders. In addition, derive the expected long-run average daily revenue if the dailyinventory cost per component is i, the cost of a replenishment order is o, the cost per day and itema order waits to be satisfied is d, and the profit of selling each item is v.Solution: For the parameters given, {Xn, n ≥ 0} is a DTMC with state space S = {1, 2, 3, 4} andtransition probability matrix

P =

0.5 0 0 0.50.3 0.5 0 0.20.2 0.3 0.5 00 0.2 0.3 0.5

.

As pii > 0, for i = 1, 2, 3, 4, the DTMC is aperiodic. Moreover, as

P 2 =

0.25 0.10 0.15 0.500.30 0.29 0.06 0.350.29 0.30 0.25 0.160.12 0.29 0.30 0.29

has all entries positive, the DTMC is irreducible.Since {Xn, n ≥ 0} is irreducible and has finite state space, there is a unique stationary proba-

bility row vector π =[π1 π2 π3 π4

], solution of π = πP and π1 = 1; i.e.,

π1 = 0.5π1 + 0.3π2 + 0.2π3

π2 = 0.5π2 + 0.3π3 + 0.2π4

π3 = 0.5π3 + 0.3π4

π4 = 0.5π1 + 0.2π2 + 0.5π4

(1.26)

21

andπ1 + π2 + π3 + π4 = 1. (1.27)

We illustrate how the solution to (1.26)-(1.27) may be obtained without the use of a computerroutine. By using successively the third, second and first equation of (1.26), we get

π3 = 0.5π3 + 0.3π4 ⇔ π3 =35π4

π2 = 0.5π2 +(

0.3× 35

+ 0.2)

π4 ⇔ π2 =1925

π4

π1 = 0.5π1 +(

0.3× 1925

+ 0.2× 35

)π4 ⇔ π1 =

87125

π4.

Using (1.27), this leads to(

87125

+1925

+35

+ 1)

π4 = 1 ⇔ π4 =125382

π4

so that,π =

[π1 π2 π3 π4

]=

[87382

95382

75382

125382

].

In addition, we can see that the fourth equation of (1.26) is also satisfied.The long-run mean inventory level is

4∑

j=1

j πj =87 + 2× 95 + 3× 75 + 4× 125

382=

501191

≈ 2.623.

In a day there is at most one order for an item which is not satisfied in the same day, whichhappens if at the beginning of the day the number of items in stock is 1 and the number of itemsordered in the day is 2. The long-run fraction of days at which there is one item ordered that isnot delivered in the same day is π1a2 = 87

382 × 0.2 = 871910 and the long-run average number of items

ordered per day is E[A1] = 0.7. Thus, the long-run fraction of ordered items that are not deliveredin the day they are ordered is 87

1910/0.7 = 871337 ≈ 6.5%.

We provide two ways to compute the expected number of days between successive replenishmentorders. Let T denote the number of days between two generic successive replenishment orders andlet T44 denote a generic return time to inventory level 4 (starting from inventory level 4). We canconclude that T is equal in distribution to

∑Nk=1 Yk with N ∼ G(0.5) and the {Yk, k ≥ 1} being

a sequence of independent and identically distributed random variables with the same distributionas T44 (why?), and N = n being independent of {Yk, k > n}. Thus, by Wald´s equation

E[T ] = E

[N∑

k=1

Yk

]= E[N ]E[T44] =

10.5

× 1π4

= 2× 382125

=764125

≈ 6.112.

An alternative way to compute the expected number of days between successive replenishmentorders is to replace the state 4 (4 items in stock) into two states: state 4r denoting that the numberof items in stock at the beginning of the day is 4 and a replenishment has been done overnight,and state 4r denoting that the number of items in stock at the beginning of the day is 4 but noreplenishment has been done overnight. Let X = {Xn, n ≥ 0} be the resulting DTMC with state

22

4 (of X) replaced by states 4r and 4r. The DTMC X has state space S = {1, 2, 3, 4r, 4r} and it iseasy to see that its transition probability matrix is

P =

0.5 0 0 0.5 00.3 0.5 0 0.2 00.2 0.3 0.5 0 00 0.2 0.3 0 0.50 0.2 0.3 0 0.5

.

As P 2 has all entries positive, X is irreducible. Thus, since X has finite state space ,it has a uniquestationary probability row vector π, solution of π = πP and π1 = 1. As, naturally, π1 = π1 = 87

382 ,π2 = π2 = 95

382 , and π3 = π3 = 75382 , we get, using the two last columns of P :

π4r = 0.5π2 + 0.2π3 ⇔ π4r =125764

π4r = 0.5π4r + 0.5π4r ⇔ π4r =125764

.

Thus π =[

87382

95382

75382

125764

125764

], and we conclude again that the expected number of days

between replenishment orders is 1/π4r = 764/125 ≈ 6.112.Although we could derive the expected long-run average daily revenue using the DTMC X, it

is simpler to do it using the DTMC X. If we consider that the cost of a replenishment order isincurred at the day the order is satisfied, then the (column) vector of expected revenue received byday in the states of X and the matrix of expected revenue received by transitions between statesof X are, respectively,

f =

−c−2c−3c

−4c− o−4c

and C =

0 0 0 1.4v − 0.4d 0v 0 0 2v 02v v 0 0 00 2v v 0 00 2v v 0 0

.

You should justify why the entries of C and f are as given. Among these entries, the one thatrequires more thought to be derived is c14r corresponding to the expected cost incurred when areplenishment of the stock is made after a day at which there was only one item in stock.

Thus, the expected long-run average daily revenue is π[P • C

]1 + πf ; i.e.,

[87382

95382

75382

125764

125764

]

0 0 0 0.7v − 0.2d 00.3v 0 0 0.4v 00.4v 0.3v 0 0 00 0.4v 0.3v 0 00 0.4v 0.3v 0 0

11111

+

−c−2c−3c

−4c− o−4c

= 0.7v − 871910

d− 501191

i− 125764

o.

This function is useful to study the effect changes in the costs d, i, and o, and selling profits v haveon the expected revenues. Note also that the value 0.7v for the expected long-run average dailyselling profits could have been simply seen to be equal to E[A1]v as all the demand for items issatisfied.

If, e.g., d = 1, v = 20, i = 0.5 and o = 10, then the expected long-run average daily revenue is21023/1910 ≈ 11.01. ¥

23

We consider next an example of an irreducible DTMC with infinite state space, in which casethe existence of a stationary distribution is not guaranteed.

Example 14 To function, a system requires a component of a given type to be functioning. Whenthe component fails it is substituted instantaneously by a new component of the same type. Assumethat the components have independent and identically distributed lifetimes with probability massfunction {pk, k ∈ IN}, and let Xn denote the age of the component in use at time n.

Determine under what conditions the DTMC X = {Xn, n ∈ IN0} has a stationary distributionand compute its stationary distribution whenever it exists.Solution: Let the random variable Y denote the lifetime of a generic component, so that pk =P(Y = k), for k ∈ IN . Moreover let {λk, k ∈ IN} denote the hazard or failure rate function of Y ;i.e.,

λk = P(Y = k|Y ≥ k) =P(Y = k)P(Y ≥ k)

=P(Y = k)∑j≥k P(Y = j)

=pk∑j≥k pj

.

If the age of the component in use at time n is k, Xn = k, then the age of the component inuse at time n + 1, Xn+1, can only be either k + 1, if the component at use does not fail before timen+1, or 1, if the component at use fails before time n+1 and it is substituted by a new one. Thus,X is a DTMC with state space IN and transition probability matrix P where

pij =

1− λi j = i + 1λi j = 10 otherwise

i.e.,

P =

λ1 1− λ1 0 0 0 . . .λ2 0 1− λ2 0 0 . . .λ3 0 0 1− λ3 0 . . .λ4 0 0 0 1− λ4 . . .. . . . . . . . . . . . . . . . . .

.

The DTMC X is irreducible. We prove this fact in the case where pk > 0 for all k ∈ IN ; theproof for the general case is a simple variant of the one presented. Note that, since pk > 0 for allk ∈ IN , 0 < λk < 1 for all n ∈ IN . Then, the irreducibility of X follows, since, for 1 ≤ i < j,

p(j−i)ij = P(Xj−i = j|X0 = i) ≥ P(X1 = i + 1, X2 = i + 2, . . . , Xj−i = j|X0 = i)

=j∏

k=i+1

P(Xk−i = k|Xk−i−1 = k − 1) =j−1∏

r=i

[1− λr] > 0

and, for 1 ≤ j < i,

p(j)ij = P(Xj = j|X0 = i) ≥ P(X1 = 1, X2 = 2, . . . , Xj = j|X0 = i)

= P(X1 = 1|X0 = i)j∏

k=2

P(Xk = k|Xk−1 = k − 1) = λi

j−1∏

r=1

[1− λr] > 0.

Thus, we conclude that X possesses at most one stationary distribution.

24

To investigate the existence of a stationary distribution, it is useful to note that

k−1∏

r=1

[1− λr] =k−1∏

r=1

[1− pr∑

j≥r pj

]=

k−1∏

r=1

∑j≥r+1 pj∑

j≥r pj=

∑

j≥k

pj = P(Y ≥ k).

The DTMC possesses a stationary distribution if and only if there exists a probability row vectorπ =

[π1 π2 . . .

]such that πP = π, i.e.,

{π1 =

∑j≥1 λj πj

πk = (1− λk−1)πk−1, k ≥ 2.

As the first equation involves all the entries of the row vector π we consider the remaining equations;using successively the equations for k = 2, 3, . . . and mathematical induction, we conclude that theylead to

πk = π1

k−1∏

r=1

[1− λr] = π1P(Y ≥ k), k ≥ 2.

Now, the normalizing condition,∑

k≥1 πk = 1, is equivalent to

∑

k≥1

πk = 1 ⇔ π1

∞∑

k=1

P(Y ≥ k) = 1 ⇔ π1E[Y ] = 1 ⇔ π1 =1

E[Y ].

If E[Y ] = +∞, then π1 = 0. This in turn implies that πk = 0 for all k ∈ IN . Thus, in the ratherfavorable situation where E[Y ] = +∞, the DTMC X does not possess a stationary distribution.Conversely, if E[Y ] < +∞, then X has a unique stationary distribution given by

πk =P(Y ≥ k)

E[Y ], k ∈ IN.

Note that this stationary distribution is very natural and could have been guessed beforehand.Moreover, if Y ∼ G(p), then the stationary distribution of X is also G(p), as in this case

πk =P(Y ≥ k)

E[Y ]=

(1− p)k−1

1/p= (1− p)k−1p, k ∈ IN. ¥

We finish this section with an example that shows that the long-run and the expected long-runproportion of time a reducible DTMC spends in a state may be different.

Example 15 Let Y = {Yn, n ≥ 0} be a DTMC with state space {0, 1}, initial probability rowvector a =

[1− p p

], 0 < p < 1, and transition probability matrix

P =[1 00 1

].

As p = P(1 = Y0 = Y1 = Y2 = . . .) = 1−P(0 = Y0 = Y1 = Y2 = . . .) it follows that

P(

limn→+∞

∑nk=0 1{Yk=j}

n + 1= 1

)= p = 1−P

(lim

n→+∞

∑nk=0 1{Yk=j}

n + 1= 0

)

i.e.,

limn→+∞

∑nk=0 1{Yk=j}

n + 1∼ Ber(p).

25

However, as

P

(n∑

k=0

1{Yk=j} = n + 1

)= p = 1−P

(n∑

k=0

1{Yk=j} = 0

)

it follows that E[∑n

k=0 1{Yk=j}]

= (n + 1)p and

limn→+∞E

[∑nk=0 1{Yk=j}

n + 1

]= p.

Note that if Yn is 1 (0) if a cash register machine is operational (not operational) at the beginningof day n and p = 90/91, then the expected long-run proportion of days the cash register machine isoperational is equal to p = 90/91, the same value as for the cash register machine of Example 11.In Example 11, 90/91 is the long-run proportion of days the cash register machine is operationalwith probability one, whereas here the long-run proportion of days the cash register machine willbe operational will be either zero or one. ¥

1.7 Time-reversibility

An irreducible DTMC X with state space S, transition probability matrix P , and stationarydistribution π is time-reversible if

πipij = πjpji (1.28)

for all i, j ∈ S. The equations (1.28) are called the detailed balance equations. They assert thatfor any pair of states (i, j), the long-run fraction of transitions from state i to state j equals thecorrespondent long-run fraction of transitions from state j to state i. They imply that the DTMClooks the same when seen forward in time as when seen backward in time. The detailed balanceequations are trivially satisfied when i = j, so they need only to be checked for pairs (i, j) withi 6= j.

An important result is that if X is an irreducible DTMC with state space S and transitionprobability matrix P , then: if there is a probability row vector π such that the equations (1.28) aresatisfied for all i, j ∈ S, then π is the stationary probability row vector of X and X is time-reversible.

This last result follows since if π is a probability row vector satisfying (1.28), then, by summingthe equations with i fixed and j taking all values in S we get

πi

∑

j∈S

pij =∑

j∈S

πjpji ⇔ πi =∑

j∈S

πjpji

for i ∈ S. Thus, π solves the (global) balance equations and so it is the stationary probability rowvector of X.

Problem 6 Given an irreducible DTMC, decide if it is time-reversible. ¥

Example 16 Consider the DTMC X of the make-to-order system with production capacity 2considered at the beginning of the chapter with the assumption that the number of orders thatarrive in a day is either 3, with probability p, or 1, with probability q = 1− p, where 0 < p < 1/2.The DTMC X has state space IN0 and, from (1.5), it follows that its transition probability matrix

26

is

P =

q p 0 0 0 . . .q 0 p 0 0 . . .0 q 0 p 0 . . .0 0 q 0 p . . .

. . . . . . . . . . . . . . . . . .

.

The DTMC X is irreducible, since, for 0 ≤ i < j,

p(j−i)ij = P(Xj−i = j|X0 = i) ≥ P(X1 = i + 1, X2 = i + 2, . . . , Xj−i = j|X0 = i)

=j∏

k=i+1

P(Xk−i = k|Xk−i−1 = k − 1) = pj−i > 0

and, for 0 ≤ j < i,

p(i−j)ij = P(Xi−j = j|X0 = i) ≥ P(X1 = i− 1, X2 = i− 2, . . . , Xi−j = j|X0 = i)

=i−1∏

k=j

P(Xi−k = k|Xi−k−1 = k + 1) = qi−j > 0.

To show that a probability row vector π satisfies (1.28) for X amounts to show that it satisfiesthe equations for |i− j| = 1; i.e., for i = 0, 1, 2, . . . and j = i+1 due to the symmetry of the balanceequations. Moreover, for i ∈ IN0 and j = i + 1, the associated detailed balance equation reads

πip = πi+1q ⇔ πi+1 =p

qπi.

Using mathematical induction, these equations lead to

πi =(

p

q

)i

π0, i ∈ IN0.

Moreover, the condition∑∞

j=0 πj = 1 necessary for π being a probability row vector leads to

∞∑

j=0

(p

q

)i

π0 = 1 ⇔ π0 =

∞∑

j=0

(p

q

)i−1

⇔ π0 = 1− p

q.

Thus, we conclude that X is a reversible DTMC with stationary probabilities

πk =(

1− p

q

)(p

q

)k

, k ∈ IN0.

Moreover, the long-run mean number of back-ordered items at the end of a day is

11− p/q

− 1 =1− p

1− 2p− 1 =

p

1− 2p

which is large when p is close to 1/2, i.e., the expected number of items ordered in a day is closeto the daily production capacity 2. ¥

Any irreducible and stationary DTMC taking values in IN0 such that the transition probabilitybetween states i and j is null whenever |i− j| > 1 is time-reversible; DTMCs of this type are calledbirth and death DTMCs. We will see more about time-reversibility and birth and death Markovchains when studying continuous time Markov chains.

27

1.8 First passage times

As a motivation for the derivations ahead consider again the setting of Example 11. Xn is equal to1 (0) if the cash register machine is operational (not operational) at the beginning of day n, andX = {Xn, n ≥ 0} is a DTMC with transition probability matrix

P =[0.10 0.900.01 0.99

].

We are interested in deriving a matrix-based method to compute the distribution function of thenumber of days the cash register machine needs to be repaired (number of days the cash registermachine needs to go from the non operational state to the operational state)

T01 = [min{n ≥ 0 : Xn = 1}|X0 = 0].

Note that if we define

Yn =

{0 X0 = X1 = X2 = . . . = Xn = 01 Xk = 1, for some k ∈ {0, 1, 2, . . . , n}

then {T01 ≤ n} = {Yn = 1|Y0 = 0}. Moreover Y = {Yn, n ≥ 0} is a DTMC with transitionprobability matrix

P (1) =[0.10 0.900 1

].

Now, as

[P (1)]n =[0.10n

∑nk=1 0.10k−10.90

0 1

]

for n ∈ IN , we conclude that P(T01 ≤ n) =∑n

k=1 0.10k−10.90; i.e., T01 ∼ G(0.90), as we couldanticipate.

Note that the DTMC Y corresponds to the modification of X for which a cash register machinestays forever operational after reaching that state, i.e., the operational state (1) is made absorbing.

As T01 ∼ G(0.90), we know that the expected number of days the cash register machine needsto go from the non operational state to the operational state is 0.90−1. However, as a motivationfor the derivation of expected times to visit sets of states of DTMCs, we illustrate how E[T01] maybe computed by conditioning on X1:

E[T01] = E[E[T01|X1]] =1∑

i=0

P[X1 = i|X0 = 0]E[T01|X1 = i]

= p00 ×E[T01|X1 = 0] + p01 ×E[T01|X1 = 1].

Now, as

E[T01|X1 = i] =

{1 + E[T01] i = 01 i = 1

we haveE[T01] = 1 + p00E[T01] ⇔ (1− p00)E[T01] = 1.

Thus, E[T01] = (1− p00)−1 = 0.90−1.In this section we will address the following general problem that we have just motivated.

28

Problem 7 Given an irreducible DTMC with finite state space, compute the distributions andexpected values of the times to reach states of a strict subset of the state space. ¥

In general, if for a finite DTMC X with state space S we want to compute the distributionfunctions of the times to reach states of a strict subset A of S departing from states outside A,

TiA = [inf{n ≥ 0 : Xn ∈ A}|X0 = i], i ∈ A

where A = S \ A, we may follow a procedure similar to the one given as example. Namely, if weorder the states of X so that the states in A appear first, the transition probability matrix P of Xmay be written in block matrix as

P =[PAA PAA

PAA PAA

]

where, e.g., PAA = [pij ]i,j∈A.If we define

Yn =

{Xn Xk ∈ A, k = 0, 1, 2, . . . , n

sA Xk ∈ A, for some k ∈ {0, 1, 2, . . . , n}then Y is a DTMC with state space A ∪ {sA} and transition probability matrix

P (A) =[PAA (I − PAA)1

0 1

]

where I is the identity matrix of rank |A| and 1 is a vector of ones of length |A|. Moreover,

P(TiA ≤ n) = P(Yn = sA|Y0 = i), i ∈ A, n ∈ IN

so that P(TiA ≤ n) is the entry (i, sA) of the nth power of the matrix P (A),

[P (A)]n =[Pn

AA

∑nk=1 P k−1

AA(I − PAA)1

0 1

]

i.e., the column vector [P(TiA ≤ n)]i∈A is given by

[P(TiA ≤ n)]i∈A =n∑

k=1

P k−1AA

(I − PAA)1.

Note that the values of the probability mass function of TiA, for i ∈ A and n ∈ IN , may be computedsimply from

P(TiA = n) = P(TiA ≤ n)−P(TiA ≤ n− 1)

with P(TiA ≤ 0) = 0. Thus, if we let f(n)iA = P(TiA = n), then the column vector f (n)

.A = [f (n)iA ]i∈A is

given byf (n).A = Pn−1

AA(I − PAA)1

for n ∈ IN , and f (n).A is the vector of first passage time probabilities into the set A in n steps.

To compute E[TiA], for i ∈ A, we condition on X1

E[TiA] = E[E[TiA|X1]] =∑

j∈S

P[X1 = j|X0 = i]E[TiA|X1 = j].

29

Now, as

E[TiA|X1 = j] =

{1 + E[TjA] j ∈ A

1 j ∈ A

we haveE[TiA] = 1 +

∑

j∈A

pij E[TjA], i ∈ A.

Thus, if we let miA = E[TiA], then, by combining the previous equations, the column vectorm.A = [E[TiA]]i∈A satisfies

m.A = 1 + PAAm.A ⇔ [I − PAA]m.A = 1 ⇔ m.A = [I − PAA]−11.

Example 17 Consider the DTMC X describing an (1, 4) inventory system given in Example 17,with state space {1, 2, 3, 4r, 4r} and transition probability matrix

P =

0.5 0 0 0.5 00.3 0.5 0 0.2 00.2 0.3 0.5 0 00 0.2 0.3 0 0.50 0.2 0.3 0 0.5

.

Compute the expected value of the number of days for a replenishment order to be made and theprobability that a replenishment order is made in at most 4 days starting with 1, 2, 3 and 4 itemsin stock.Solution: Here the set A of interest is a singular set {4r} and A = {1, 2, 3, 4r}. Moreover theassociated transition probability matrix with state 4r made absorbing and the order of states 4rand 4r interchanged, so that the states become ordered as 1, 2, 3, 4r, 4r, is

P (4r) =

0.5 0 0 0 0.50.3 0.5 0 0 0.20.2 0.3 0.5 0 00 0.2 0.3 0.5 00 0 0 0 1

.

Thus,

PAA =

0.5 0 0 00.3 0.5 0 00.2 0.3 0.5 00 0.2 0.3 0.5

, (I − PAA)1 =

0.50.200

and the vector of expected values of the number of days for a replenishment order to be made is

m1 4r

m2 4r

m3 4r

m4r 4r

=

E[T1 4r]E[T2 4r]E[T3 4r]E[T4r 4r]

= (I − PAA)−11 =

0.5 0 0 0−0.3 0.5 0 0−0.2 −0.3 0.5 0

0 −0.2 −0.3 0.5

−1

1111

=

23.24.726.112

.

Thus, the expected value of the number of days for a replenishment order to be made starting with1, 2, 3 and 4 items in stock is 2, 3.2, 4.72 and 6.112, respectively.

30

As

[P (4r)]4 =

0.0625 0 0 0 0.93750.15 0.0625 0 0 0.78750.235 0.15 0.0625 0 0.55250.234 0.235 0.15 0.0625 0.3185

0 0 0 0 1

it follows that

P(T1 4r ≤ 4)P(T2 4r ≤ 4)P(T3 4r ≤ 4)P(T4r 4r ≤ 4)

=

0.93750.78750.55250.3185

.

Thus, the probability that a replenishment order is made in at most 4 days starting with 1, 2, 3and 4 items in stock is 0.9375, 0.7875, 0.5525 and 0.3185, respectively. Note that there exist largedifferences between the computed probabilities. ¥

Example 18 For the telecommunications switch of Example 6, compute the expected value of thenumber of time slots needed for the buffer to have at least 9 packets (for the first time) at the endof the time slot and the probabilities that the buffer contains at least 9 packets at or before timeslots 10 and 100 starting with 0, 1, 2, . . . , 8 packets in the buffer.Solution: Here the set A of interest is {9, 10} and A = {0, 1, 2, . . . , 8}. Moreover the associatedtransition probability matrix with states 9 and 10 amalgamated into state 9+ and state 9+ madeabsorbing has state space {0, 1, 2, . . . , 8, 9+} and transition probability matrix

P ({9, 10}) =

a0 a1 a2 a3 a4 a5 0 0 0 0a0 a1 a2 a3 a4 a5 0 0 0 00 a0 a1 a2 a3 a4 a5 0 0 00 0 a0 a1 a2 a3 a4 a5 0 00 0 0 a0 a1 a2 a3 a4 a5 00 0 0 0 a0 a1 a2 a3 a4 a5

0 0 0 0 0 a0 a1 a2 a3 a4 + a5

0 0 0 0 0 0 a0 a1 a2∑5

i=3 ai

0 0 0 0 0 0 0 a0 a1∑5

i=2 ai

0 0 0 0 0 0 0 0 0 1

where the ak’s are the binomial probabilities with parameters 5 and 0.2,

ak =

0.85 k = 00.84 k = 10.4× 0.83 k = 20.08× 0.82 k = 34× 0.24 k = 40.25 k = 50 otherwise

.

31

Thus, since [E[TiA]]i∈A = (I − PAA)−11 and [P(TiA ≤ n)]i∈A =∑n

k=1 P k−1AA

(I − PAA)1, we get

E[T0A]E[T1A]E[T2A]E[T3A]E[T4A]E[T5A]E[T6A]E[T7A]E[T8A]

=

94.5594.5592.0587.0579.5569.5557.0542.0425.02

,

P[T0A ≤ 10]P[T1A ≤ 10]P[T2A ≤ 10]P[T3A ≤ 10]P[T4A ≤ 10]P[T5A ≤ 10]P[T6A ≤ 10]P[T7A ≤ 10]P[T8A ≤ 10]

=

0.0100.0100.0200.0420.0860.1610.2820.4550.674

,

P[T0A ≤ 100]P[T1A ≤ 100]P[T2A ≤ 100]P[T3A ≤ 100]P[T4A ≤ 100]P[T5A ≤ 100]P[T6A ≤ 100]P[T7A ≤ 100]P[T8A ≤ 100]

=

0.6560.6560.6670.6890.7210.7620.8100.8650.924

. ¥

We note that the procedure used here could be easily generalized to compute expected costsincurred strictly before visiting a state in the subset A of the finite state space S of a DTMC X.If, e.g., we let Yk denote the cost incurred at time k and cj = E[Yk|Xk = j], for j ∈ A, with cj

being finite, then proceeding as before, we would conclude that the column vector of expected costsincurred strictly before visiting a state in the subset A is given by

[E

[TiA−1∑

k=0

Yk

]]

i∈A

= (I − PAA)−1c

with c = [ci]i∈A being the column vector of expected costs incurred by unit time spent in each stateof A. In this respect see the Conceptual Problem 5.18 of the textbook.

1.9 Final remarks

The chapter on DTMCs is very important and it should be studied in a rather comprehensiveway as it contains most of the main aspects of stochastic reasoning that will be used along thecourse. A special emphasis has been given to the presentation of algorithmic approaches thatpermit to solve the problems that were identified along the chapter. This is specially relevantas almost all industrial problems require the use of computer programs to obtain their solution.The formulas that we derived in the chapter to compute the most important performance measuresassociated with DTMCs are simple and are easy computable with a wide range of computer packages(Mathematica, MATLAB, Maple, etc.) and (with the exception of the distribution function of firstpassage times) are implemented in MAXIM.

An alternative way to implement the computation of the stationary distribution of a DTMCwith finite state space that avoids the non-invertibility of the transition probability matrix is nextdescribed.

Suppose that X is a DTMC with state space {1, 2, . . . , k} and transition probability matrixP = [pij ]i,j=1,2,...,k and let π be the row vector

[π1 π2 . . . πk

]. Then the system πP = π with∑k

j=1 πj = 1 is equivalent to the non-singular system of linear equations

1 1 1 1 . . . 1p12 p22 − 1 p32 p42 . . . pk2

p13 p23 p33 − 1 p43 . . . pk3

p14 p24 p34 p44 − 1 . . . pk4

. . . . . . . . . . . . . . . . . .p1k p2k p3k p4k . . . pkk − 1

π1

π2

π3

π4

. . .πk

=

1000. . .0

.

32

Note that the matrix of coefficients of the previous linear system of equations may be obtained fromthe transition probability matrix P by: (1) subtracting one from each value in the main diagonal ofthe matrix; (2) changing all the values in the first column to ones; and (3) transposing the resultingmatrix.

We have noticed that the amount of time a DTMC with state space S and transition probabilitymatrix P = [pij ]i,j∈S stays in state k in a visit to the state has geometric distribution with successprobability

1− pkk =∑

i6=k

pki = P(Xn+1 6= k|Xn = k)

i.e., the probability of a transition out from state k. Moreover, if 1− pkk > 0, i.e., it is possible toleave state k, then the probability that the state visited when leaving state k is j is

pkj = P(Xn+1 = j|Xn = k,Xn+1 6= k) =P(Xn+1 = j|Xn = k)P(Xn+1 6= k|Xn = k)

=pkj∑i6=k pki

=pkj

1− pkk.

for j 6= k and pkk = 0. As pkj = (1−pkk)pkj for all j, k ∈ S with j 6= k (this also holds if 1−pkk = 0,in which case pkj = 0 for all j 6= k, so that we may choose pkj arbitrarily – a common choice beingpkj = 0 for j 6= k) we see that the transition probabilities of the DTMC X are characterized by thevector and matrix of parameters

q = [1− pkk]k∈S and P = [pkj ]k,j∈S .

q is the column vector of (one-step) probabilities of leaving the states of X and P is the (one-step)transition probability matrix of X conditional to a transition out of the current state taking place.In the next chapter we will see the analogy of this interpretation of a DTMC with one of the mostcommon interpretations for continuous time Markov chains.

1.10 Case study: The cash register machine contract

Mr. Bright, a young man was hired by Uniforms Inc. just after finishing his MBA at one of the topmanagement schools in the world. At his first meeting with Mr. Golden Eye, the Chief Executive,he told him that he was expecting him to be a very important addition to the team (of managers)of Uniforms Inc. and to give a strong contribute to increase the company´s sales volume.

Mr. Golden Eye revealed that he had on hands a very important decision about a contractto buy 1000 cash register machines to equip the new stores that were about to open. Moreoverhe told Mr. Bright that we was considering buying the machines to Registers King Inc., a majorproducer of these machines. The reason was that the top managers of Register King Inc. hadassured him that they could sell to Uniforms Inc. at $ 2000 machines that had a daily reliabilityof 99% and only on 10% of the days in which they were being fixed they would need additionaldays for the repairing to be completed. Moreover, Registers King Inc. would collect the machinesin need of repairing at the beginning of each business day and return repaired machines at the endof the business day, with no additional costs for 12 years (the amount of time Mr. Golden Eye wasexpecting that style of cash register machines would be used at Uniforms Inc.).

As the acquisition of this cash register machines would result in a contract of about 2 milliondollars and as a sign of confidence in Mr. Bright, Mr. Golden Eye gave Mr. Bright as his firstmission the project of obtaining at a “right price” the proportion of days one should expect amachine produced by Registers King Inc. would be functioning in the first 12 years of use.

33

Mr. Bright started to work immediately on the project by contacting 3 of the major consultingcompanies (Everything That Is Possible, Impossible Made Possible, and Right Solutions) to askfor quotes for deriving the expected proportion of days a machine produced by Registers King Inc.would be functioning in the first 12 years of use.

Everything That Is Possible answered back in two days. They said they had put their bestpeople working to produce a quote and that they had been conclusive: they couldn’t´t (as wellas any other company in the market) carry out that project; it was just impossible! Mr. Brightstarted to be concerned with the project!

Impossible Made Possible took four days to answer back. They said they had put their bestsenior people working to produce a quote and that they had reached the following conclusions: (1)the project was really big and they had not the necessary infrastructure to carry it; (2) due totheir privileged contacts they could however carry out the project by involving one of the majoruniversities in the country and using a supercomputer. The estimated cost to carry out the projectwas 200 million dollars. Moreover, it could not be finished before two years and they would needan advance paiement of at least 50%. Mr. Bright was now really concerned with the project, if itcould be carried out it seemed as it would take too long and be too expensive!

Mr. Bright was thus very pleased when he was contacted by Right Solutions one week later.They told him that Uniforms Inc. should not go for expensive consulting companies that would askmillions of dollars to carry out the project without being able to guarantee delivery times. Theycould carry out the project at the right price of just 300 thousand dollars. They had recruited thebest MBA graduates and it had really paid off. These smart MBA graduates had been workingto produce a quote and based on the CK principle, they had devised a very sophisticated wayto compute the expected number of days a machine produced by Registers King Inc. would befunctioning in the first 12 years of use. This approach would be carried out in 144 desktop computersin at most two weeks. Each of them would compute the expected number of days a machine wouldbe functioning in one of the 144 months of the first 12 years of use, so that Uniforms Inc. wouldbe able to get this additional information at no additional cost.

Mr. Bright, who had been thinking about what he had learned in the MBA, concluded that itwas still too expensive to pay 300 thousand dollars for the project as this was roughly the acquisitionvalue of 150 cash register machines. A better alternative would probably be to buy 150 additionalmachines to substitute machines under repair. Thus he decided to contact Mr. Startup who was along time friend of him and had left a promising PhD in IE to start a small IE company, called IEReally Works!

Since Mr. Bright called IE Really Works early in the morning, Mr. Startup was still not atwork. Mr. Bright´s call was answered by Simple, a smart senior IE student that was workingbecause he needed some bucks to finish up his undergraduate studies. Simple was surprised why anMBA graduate would need to contract a company to solve that type of question, but as he knewthat the financial situation of IE Really Works was not famous he decided to give Mr. Wrightexpensive quotes.

Namely, Simple told Mr. Wright that he could give him to choices: (1) the precise value of theexpected proportion of days a machine would be functioning in the first 12 years of use, and at noextra cost the probability that the machine would be working in each day of the first 12 years, and(2) a very good estimate of the expected proportion of days a machine would be functioning in thefirst 12 years of use, but with no information about the probability the machine would be workingin each day of the first 12 years. For the first choice IE Really Works would charge the standardrate for a one-day job of the full company´s team (i.e., Simple, since Mr. Startup was always busy

34

trying to sell IE solutions – a fact that Simple naturally did not reveal to Mr. Bright): 10 thousanddollars. For the second choice the standard one-hour rate would be applied: 2 thousand dollars.

Mr. Bright thought he could not loose the opportunity to make his first decision as a realmanager; thus, he decided on the spot to go for the second choice of IE Really Works. By the endof the day he had at his desk the magical number he was looking for: 0.98901099.

Mr. Bright had to use all of his skills to persuade Mr. Golden Eye´s secretary to schedule himan appointment with Mr. Golden Eye for the next day. He was really proud of himself when in thenext day he went to Mr. Golden Eye´s office. Mr. Golden Eye was very friendly and said that hewanted to know everything Mr. Bright had done to obtain the value 0.98901099 for the expectedproportion of days a machine would be functioning in the first 12 years of use for just 2 thousanddollars, which Mr. Bright did.

At the end of the appointment, Mr. Golden Eye congratulated Mr. Bright for his work to cutthe costs of the project. He reinforced the congratulations with his appraisal for him choosing tojust get an estimate of the expected proportion of days a machine would be functioning in the first12 years of use, as Uniforms Inc. just needed a good estimate anyway. It was thus really a bigsurprise to Mr. Wright when Mr. Golden Eye finished the appointment by telling him: “I am reallysorry that we will have to cut in your salary bonus the 2 thousand dollars you have spent in theproject as, for a first project in the company, you really did a good job, but this is our policy!”

We next provide possible explanations for the attitude of Mr. Golden Eye, by increasing orderof likelihood. We warn that there may be other explanations as good or better than the ones given.In the following we assume always that π1 (π0) is the long-run fraction (or expected fraction) ofdays a cash register machine is operational.Explanation 1: Mr. Golden Eye had taken an advanced financial modelling course during hisMBA degree and, thus, knew that for a two-state DTMC the long-run fraction of time the DTMCspends in a state equals the transition probability into the state (from the other state) divided bythe sum of the transition probabilities between different states. Accordingly, he had concluded thatthe long-run fraction of days a cash register machine would be operational was 0.90/(0.01 + 0.90)or 90/91.Explanation 2: Mr. Golden Eye had a good reasoning and had some knowledge of basic prob-ability distributions, so he knew the geometric distribution. He reasoned that between the daysat which two successive repairs of the cash register machine would be finished one should expectto have 0.01−1 days with the cash register machine being operational and 0.90−1 days with it be-ing non operational. Thus the long-run expected proportion of days the cash register would beoperational had to be 0.01−1/(0.01−1 + 0.90−1) or 90/91.

Mr. Golden Eye used a reasoning that is similar to a result for alternating renewal processes (oron-off processes) that we will see later on in this course. Note that 0.01−1 (0.90−1) is the expectedvalue of a geometric random variable with parameter 0.01 (0.90).Explanation 3: Mr. Golden Eye had a good reasoning, so he knew perfectly than in the long-runthe rate at which a cash register machine would get non-operational would be equal to the rateat which it would get repaired; by equating π1 × 0.01 = π0 × 0.90 we concluded that the long-runfraction of days the cash register machine would be operational would be 90 times the long-runfraction of days the cash register machine would not be operational. Thus, he was sure that thelong-run fraction of days the cash register machine would be operational would be 90/91.

Note the similarity of Mr. Golden Eye’s reasoning with the concept of time-reversibility (thisidea really works as any two-state irreducible DTMC is time-reversible!)

35

Explanation 4: Mr. Golden Eye had a good reasoning and had some knowledge of basic proba-bility distributions, so he knew the geometric distribution. He reasoned that the expected numberof consecutive days a cash register would be repairing would be 0.90−1 (the expected value of ageometric random variable with success probability 0.90) and that the long-run rate at which newperiods with the cash register machine being repaired would be initiated could be a most 0.01 (theprobability of a cash register machine making a transition from operational to non-operational ona given day). As a consequence the long-run fraction of days a cash register machine would beoperational would be at least 1− 0.01/0.90 or 89/90 (note that this lower bound differs from 90/91by about 0.01%, a difference that is not likely to be significant in practice).Explanation 5: Mr. Golden Eye went on to tell Mr. Wright “The fact is that you have not takeninto consideration the most important factor in management: that is, the human/business relationsfactor. In fact, I am a long time friend of Registers King Inc.´s Chief Executive and we are aboutto close with them a very important deal to produce the uniforms of Registers King Inc. for thenext 10 years. Thus, I closed the deal to buy the 1000 cash register machines to Cash RegistersInc. in the lunch I had with their Chief Executive yesterday!”

36

Chapter 2

Continuous Time Markov Chains

In this chapter we will study the evolution of stochastic systems whose state changes may occur atany instant of time and not just at integer time points. We are thus lead to the study of systemsin continuous time as apposed to the discrete-time systems we have studied in Chapter 1.

Recall three examples of stochastic systems given at the beginning of Chapter 1: the DowJones stock index, the arrival of orders for new subscriptions of telephone lines; the production ofmanufactured items by a machine with random production times (and/or with random supply ofparts). These systems are in fact mostly suited to be modelled using continuous time processes.

Continuous time Markov chains (CTMCs) are the special cases of stochastic processes in con-tinuous time that: take values on a finite or countable set, i.e. have discrete state space, and inaddition possess the Markov property of the future evolution of the process being independent ofits past evolution given that its present state is known.

As in Section 1.1 a detailed motivation for the study of processes possessing the Markov property(i.e., Markov processes) was given, we will proceed directly to the definition of CTMCs (you maywant to read again Section 1.1 at this point) after we recall some properties of the exponentialdistribution and sequences of exponential random variables which are useful for the interpretationof CTMCs. Some of these properties are, in addition, relevant to deduce properties of Poissonprocesses, which we illustrate.

2.1 Properties of the exponential distribution and the Poissonprocess

This main objective of this section is to recall some results you know and derive also new ones foruse in the rest of the course. The section may be seen as accessory, although of high importance.The problem we address in this section may be generally phrased in the following way.

Problem 8 Compute quantities associated to exponential random variables, the Poisson process,and transformations resulting from them.

The exponential distribution is a positive distribution widely used in practice, among othersto model lifetimes of components. A positive random variable X is said to have exponentialdistribution with parameter λ > 0, X ∼ Exp(λ), if it has probability density function

fX(x) = λe−λx, x > 0 (2.1)

37

and, since X is a positive random variable, fX(x) = 0 for x ≤ 0. Its (cumulative) distributionfunction is

FX(x) =

{0 x ≤ 01− e−λx x > 0

. (2.2)

In particular, its survival function (complement of the distribution), F , satisfies F (x) = e−λx

for x > 0 – thus, the survival function is an exponential function with negative exponent, whichexplains in part why the exponential distribution is also called negative exponential distribution.

An important property of the exponential distribution is its lack of memory property, whichturns out to be an immediate consequence of the exponential form of its survival function. Acontinuous positive random variable Y is said to possess the lack of memory property if

P(Y > s + t) = P(Y > s)P(Y > t), for all s, t ≥ 0

[for an Exp(λ) random variable this reads e−λ(s+t) = e−λse−λt for all s, t ≥ 0] or, equivalently,

P(Y > s + t|Y > s) = P(Y > t), for all s, t ≥ 0

i.e., the distribution of the residual lifetime of Y at age s, (Y − s|Y > s), has the same distributionas Y itself. Thus, a component whose lifetime possesses the lack of memory property does not agewith time, i.e., an used component is as good as a new one. In fact, as seen in Exercise 3 of theReview Exercises (which you should review), the exponential distribution is the unique continuousdistribution that possesses the lack of memory property.

Note that, since the Taylor series expansion of e−λx is

e−λx = 1− λx +(λx)2

2!− (λx)3

3!− (λx)4

4!+ . . .

the instantaneous failure rate of X ∼ Exp(λ) is

lim∆t→0+

P(X ≤ t + ∆t|X > t)∆t

= lim∆t→0+

1− e−λ∆t

∆t= lim

∆t→0+

λ∆t− (λ∆t)2

2! + (λ∆t)3

3! − . . .

∆t= λ. (2.3)

The the exponential distribution has constant instantaneous failure rate which is equal to theparameter of the distribution. Note that from (2.3) we have

P(t < X ≤ t + ∆t|X > t) ≈ λ∆t, for ∆t small

i.e., given that an exponential random variable has survived until time t, the probability that it failswithin a small time interval of length ∆t is roughly ∆t times the rate of the exponential randomvariable.

In order to compute the moments of the exponential distribution, we first note if X ∼ Exp(λ),then E[X0] = 1 and for n ∈ IN ,

E[Xn] =∫ ∞

0xnλe−λxdx

=(−xne−λx

]∞0

+∫ ∞

0nxn−1e−λxdx by integration by parts

= (0− 0) +n

λ

∫ ∞

0xn−1λe−λxdx by rearranging terms

=n

λE

[Xn−1

].

38

As a consequence, and using induction, we conclude that

E [Xn] =n!λn

, n ∈ IN0.

In particular, E[X] = 1/λ and Var[X] = E[(X −E[X])2] = E[X2]−E2[X] = 2/λ2 − 1/λ2 = 1/λ2.We next look at properties of sums of independent and identically distributed exponential

random variables, i.e., the distribution of Sn = X1 + X2 + . . . + Xn with {Xi}i≥1iid∼ Exp(λ),

which corresponds to the Erlang distribution with parameters n ∈ IN and λ > 0 – denoted byErlang(n, λ). In particular, Sn has expected value n/λ and variance n/λ2, since

E[Sn] = E

[n∑

i=1

Xi

]=

n∑

i=1

E[Xi] = n×E[X1] =n

λ

and, as the random variables Xi are independent,

Var[Sn] = Var

[n∑

i=1

Xi

]=

n∑

i=1

Var[Xi] = n×Var[X1] =n

λ2.

The parameter n is usually called the number of phases of the Erlang distribution and λ its rate.The Erlang distribution, which is a particular case of the gamma family of distributions, has veryimportant applications with emphasis in queueing networks.

Note that if the random variables Xi denote the (inter-event) times between the occurrence ofthe (i − 1)-th and i-th events in a Poisson process (with rate λ), {N(t), t ≥ 0}, then Sn denotesthe time of the occurrence of the n-th event in the Poisson process. In this case N(t), the numberof events that occur in the (time) interval [0, t] is given by

N(t) = max{n ∈ IN0 : Sn ≤ t} (2.4)

for t ≥ 0, where S0 = 0. In the following we let PP (λ) stand for a Poisson process with rate λ, sothat {N(t), t ≥ 0} ∼ PP (λ).

We now proceed to characterize the Erlang(n, λ) distribution. Using induction it follows thatSn has probability density function

fSn(s) =λ(λs)n−1e−λs

(n− 1)!, s > 0 (2.5)

and fSn(s) = 0 for s ≤ 0. To show this last relation, first see that it is true for n = 1, in which caseit is just the probability density function of an Exp(λ). Moreover, if we assume that (2.5) is truefor n = k, for some k ∈ IN , then

fSk+1(s) =

∫ s

0fSk

(x)fXk+1(s− x)dx

=∫ s

0

λ(λx)k−1e−λx

(k − 1)!λe−λ(s−x)dx by induction hypothesis

= λe−λs

∫ s

0

λ(λx)k−1

(k − 1)!dx rearranging terms

= λe−λs

((λx)k

k!

]s

0

= λe−λs

((λs)k

k!− 0

)=

λ(λs)ke−λs

k!

39

for s > 0, and fSk+1(s) = 0 for s ≤ 0. Thus, (2.5) is true for n = k + 1.

This shows that (2.5) is in fact the probability density function of the sum on n independent andidentically distributed random variables with Exp(λ) distribution, for any n ∈ IN . Note that theexpected value and variance of the Erlang(n, λ) distribution could alternatively have been derivedusing (2.5).

Similarly, as Xiiid∼ Exp(λ), we have

P(S1 > t) =∫ ∞

tλe−λxdx =

(e−λx

]∞t

= e−λt, t ≥ 0

and, moreover, using mathematical induction, we can conclude that

P(Sn > t) =n−1∑

k=0

e−λt (λt)k

k!(2.6)

for all n ∈ IN and t ≥ 0. To conclude this suppose that the last equation is true for some n ∈ IN

and note that Sn+1 − S1 =∑n+1

k=2 Xid=

∑nk=1 Xi = Sn; then

P(Sn+1 > t) = P(S1 > t) +∫ t

0fS1(x)P(Sn+1 > t|S1 = x)dx by the total probability law

= P(X1 > t) +∫ t

0fX1(x)P(Sn+1 − S1 > t− x)dx

= e−λt +∫ t

0λe−λxP(Sn > t− x)dx since Sn+1 − S1

d= Sn

= e−λt +∫ t

0λe−λx

n−1∑

k=0

e−λ(t−x) [λ(t− x)]k

k!dx by induction hypothesis

= e−λt

[1 +

n−1∑

k=0

∫ t

0λ

[λ(t− x)]k

k!dx

]by rearranging terms

= e−λt

[1 +

n−1∑

k=0

(− [λ(t− x)]k+1

(k + 1)!

]t

0

]

= e−λt

[1 +

n−1∑

k=0

(λt)k+1

(k + 1)!

]

=n∑

k=0

e−λt (λt)k

k!.

Thus the distribution function of Sn ∼ Erlang(nλ) is

P(Sn ≤ t) = 1−n−1∑

k=0

e−λt (λt)k

k!=

∞∑

k=n

e−λt (λt)k

k!(2.7)

for t ≥ 0 and P(Sn ≤ t) = 0 for t < 0. You may try to conclude (2.7) from (2.5) using integrationby parts. Note that the summands appearing in (2.7) are Poisson probabilities associated to aP (λt) random variable, and (2.7) is equivalent to

P(Erlang(n, λ) ≤ t) = P(P (λt) ≥ n)

40

for all n ∈ IN and t ≥ 0, where Erlang(n, λ) and P (λt) stand for generic random variables withErlang(n, λ) and P (λt) distributions, respectively. Thus, values of the distribution function of anErlang random variable may be obtained using values of the Poisson distribution function.

Let us recall the interpretation of the random variable Sn ∼ Erlang(n, λ) standing for the timeof occurrence of the n-th event in a Poisson process with rate λ, {N(t), t ≥ 0}. As,

{Sn ≤ t} = {N(t) ≥ n}, for all n ∈ IN, t ≥ 0

i.e., the n-th event of the Poisson process {N(t), t ≥ 0} occurs until time t if and only if n or moreevents are observed until time t, it follows that

P(Sn ≤ t) = P(N(t) ≥ n), for all n ∈ IN, t ≥ 0

In view of (2.7), this leads to

P(N(t) ≥ n) =∞∑

k=n

e−λt (λt)k

k!, for all n ∈ IN, t ≥ 0

and so N(t) ∼ P (λt).Thus, we have concluded that in a Poisson process with rate λ, {N(t), t ≥ 0}, the number of

events that occur in the (time) interval [0, t], N(t), has P (λt) distribution, i.e.,

P(N(t) = k) = e−λt (λt)k

k!(2.8)

for all k ∈ IN0, t ≥ 0, which justifies that the process is called Poisson process.

Example 19 Orders for laptops arrive at a computer manufacturing facility according to a Poissonprocess with rate 0.5/h. Compute the probability that more than 7 orders for laptops are receivedover a 4h period and the probability that the third laptop order arrives in the second hour ofoperation.

Solution: Let {N(t), t ≥ 0} ∼ PP (0.5) be the counting process of laptop order arrivals (with timein hours). Due to the lack of memory property of the exponential distribution, the probability thatmore than 7 orders for laptops are received over a 4h period depends on the specific time intervalonly through its length (4h) and is equal to

P(N(4) > 7) = 1−P(N(4) ≤ 7)

= 1−7∑

k=0

e2 2k

k!since N(4) ∼ P (0.5× 4), i.e., N(4) ∼ P (2)

≈ 1− 0.9989 = 0.11%.

Let Sn denote the time (h) of the arrival of the n-th laptop order. Then, the probability thatthe third laptop order arrives in the second hour of operation is

P(1 < S3 ≤ 2) = P(S3 ≤ 2)−P(S3 ≤ 1)= P(N(2) ≥ 3)−P(N(1) ≥ 3)= [1−P(N(2) ≤ 2)]− [1−P(N(1) ≤ 2)]

=

[1−

2∑

k=0

e−1 1k!

]−

[1−

2∑

k=0

e−0.5 0.5k

k!

]since N(1) ∼ P (0.5) and N(2) ∼ P (1)

41

=2∑

k=0

e−0.5 0.5k

k!−

2∑

k=0

e−1 1k!

≈ 0.9856− 0.9197 = 0.0659 = 6.59%. ¥

This concludes our analysis of sums of independent and identically distributed exponentialrandom variables with a fixed number of summands, which lead us to the Erlang distributionand the Poisson process and distribution. Next we look at particular sums of independent andidentically distributed exponential random variables with a random number of summands; namely,a geometric distributed number of summands.

If again {Xi}i≥1iid∼ Exp(λ), M ∼ G(p), and M is independent of the sequence {Xi}i≥1, then

the random sum SM = X1 +X2 + . . .+XM has Exp(λp) distribution. This result follows since thesurvival function of SM is given by

P(SM > t) =∞∑

n=1

P(M = n)P(SM > t|M = n) by the total probability law

=∞∑

n=1

P(M = n)P(Sn > t)

=∞∑

n=1

(1− p)n−1pn−1∑

k=0

e−λt (λt)k

k!by (2.6)

= e−λt∞∑

k=0

(λt)k

k!

∞∑

n=k+1

(1− p)n−1p by rearranging terms

= e−λt∞∑

k=0

(λt)k

k!(1− p)k by summing the geometric series

= e−λt∞∑

k=0

(λ(1− p)t)k

k!by rearranging terms

= e−λteλ(1−p)t

= e−λt+λ(1−p)t

= e−(λp)t

for t ≥ 0.

Thus, if {Xi}i≥1iid∼ Exp(λ) and M ∼ G(p) is independent of {Xi}i≥1, then

SM =M∑

n=1

Xi ∼ Exp(λp). (2.9)

This property has a nice consequence for Poisson processes. For suppose that each event of{N(t), t ≥ 0} ∼ PP (λ) is registered with probability p, independently of other events, and letR(t) denote the number of registered events in the interval [0, t]. Then, the inter-event times of{R(t), t ≥ 0} are distributed like SM and moreover they are independent, by the lack of memoryproperty of the exponential distribution. But, since SM ∼ Exp(λp), this implies that

{R(t), t ≥ 0} ∼ PP (λp). (2.10)

42

{R(t), t ≥ 0} is said to be an (independent) Bernoulli thinning with probability p of the Poissonprocess {N(t), t ≥ 0} because {R(t), t ≥ 0} retains each event of {N(t), t ≥ 0} with probability pand discards it with probability 1− p. Thus, an alternative way to define {R(t), t ≥ 0} would bethrough

R(t) =N(t)∑

i=1

Yi (2.11)

for t ≥ 0, with {Yi}i≥1iid∼ Ber(p). Note that, since N(t) ∼ P (λt), the fact R(t) ∼ P (λpt) would

also follow from Exercise 4 (b) of the Review Exercises, which shows that an independent Bernoullithinning with probability p of a Poisson random variable with parameter µ has Poisson distributionwith parameter µp.

By the same reason, the counting process of unregistered events is also a Poisson process, namely

{N(t)−R(t), t ≥ 0} ∼ PP (λ(1− p)). (2.12)

Taking into account Exercise 4 (b) of the Review Exercises it is natural to guess that the countingprocesses of registered and unregistered events, {R(t), t ≥ 0} and {N(t)−R(t), t ≥ 0}, are in factindependent processes, which is indeed true. However, a different analysis needs to be used in orderto prove such result.

Example 20 Suppose that in Example 19 each order is correctly processed (i.e., the specificationsof the laptop ordered are correctly registered) with probability 0.98, independently of other orders.Compute the probability that at least one order is not correctly processed in the first 24h ofoperation.

Solution: Let {R(t), t ≥ 0} be the counting process of laptop order arrivals (with time in hours)that are not correctly processed. Since each order is not correctly processed with probability1−0.98 = 0.02, independently of other orders, and the process of laptop order arrivals is a PP (0.5),{R(t), t ≥ 0} ∼ PP (0.5 × 0.02). Thus, the probability that at least one order is not processedcorrectly in the first 24h of operation is

P(R(24) ≥ 1) = 1−P(R(24) = 0)

= 1− e−24×0.5×0.02 since R(24) ∼ P (24× 0.5× 0.02)

= 1− e−0.24 ≈ 1− 0.7866 = 0.2134. ¥

We now consider the situation where we have a fixed number of exponential random variablescompeting (this may correspond, e.g., to looking for the first failure of a number of items witheach one having an exponentially distributed lifetime), with the exponential random variables nothaving necessarily the same parameter.

Thus, suppose that Xi, i = 1, 2, . . . , n, are independent random variables and Xi ∼ Exp(λi),and define

Z = min(X1, X2, . . . , Xn) and W = min{1 ≤ i ≤ n : Z = Xi}.The random variable Z corresponds to the smallest of the random variables X1, X2, . . . , Xn and Wto the index of the smallest of these random variables (note that the probability of ties among therandom variables X1, X2, . . . , Xn is zero since these are independent continuous random variables).For the analysis of the joint distribution of (Z, W ) for the case where n = 2 see the Exercise 5 of

43

the Review Exercises; similarly, for the marginal distribution of Z for arbitrary n see Exercise 6(a) of the Review Exercises.

Let fXi(.) denote the probability density function of Xi, i = 1, 2, . . . , n. We characterize thejoint distribution of (Z, W ) through the probabilities

P(Z > t,W = j) =∫ ∞

tfXj (x)P(Xi > x, 1 ≤ i ≤ n, i 6= j) dx by the total probability law

=∫ ∞

tfXj (x)

∏

i 6=j

P(Xi > x) dx by the independence of the Xi’s

=∫ ∞

tλje

−λjt∏

i6=j

e−λix dx since Xk ∼ Exp(λk), 1 ≤ k ≤ n

= λj

∫ ∞

t

n∏

i=1

e−λix dx by rearranging terms

= λj

∫ ∞

te−Pn

i=1 λix dx

= λj

(−e−

Pni=1 λix

∑ni=1 λi

]∞

t

= λj

(0 +

e−Pn

i=1 λit

∑ni=1 λi

)

= e−(Pn

i=1 λi)tλj∑ni=1 λi

by rearranging terms

for t ≥ 0 and j ∈ {1, 2, . . . , n}.This shows that

P(Z > t, W = j) = e−(Pn

i=1 λi)tλj∑ni=1 λi

= P(Z > t)P(W = j) (2.13)

for all t ≥ 0 and j ∈ {1, 2, . . . , n}, and Z and W are independent. This follows since

P(Z > t) =n∑

j=1

P(Z > t, W = j) = e−(Pn

i=1 λi)t, t ≥ 0

so that Z ∼ Exp(∑n

i=1 λi), and, moreover,

P(W = j) = P(Z > 0,W = j) =λj∑ni=1 λi

, j ∈ {1, 2, . . . , n}.

Thus, the minimum of independent exponential random variables has exponential distributionwith parameter equal to the sum of the rates of the intervenient exponential random variables. Inaddition, the probability that each of these random variable is equal to their minimum is equal tothe ratio of its rate and the sum of the rates of all random variables. Moreover, due to the lack ofmemory of the exponential distribution, the random variables (Xi − Z|Xi > Z) are independentand have Exp(λi) distribution.

This property leads to the conclusion that the inter-event times of the counting process resultingfrom the addition of n independent Poisson processes, {Ni(t), t ≥ 0} ∼ PP (λi), i = 1, 2, . . . , n, are

44

independent random variables with Exp(∑n

i=1 λi) distribution, so that{

n∑

i=1

Ni(t), t ≥ 0

}∼ PP

(n∑

i=1

λi

). (2.14)

Moreover, each event of the aggregated counting process {∑ni=1 Ni(t), t ≥ 0} is originated from the

j-th Poisson process, {Nj(t), t ≥ 0}, with probability λj/∑n

i=1 λi, independently of other events.Note that (2.14) strengthens the property, shown in Exercise 1 (a) of the Review Exercises, that

the sum of independent Poisson random variables is a Poisson random variable with parameter equalto the sum of the parameters of the summed Poisson random variables.

The converse of (2.14) is the result that if N = {N(t), t ≥ 0} ∼ PP (λ) and each event of N isof type i, 1 ≤ i ≤ n, and only of type i with probability pi, independently of other events, and welet Ni(t) denote the number of events of type i observed on the time interval [0, t], then

Ni = {Ni(t), t ≥ 0} ∼ PP (λpi) (2.15)

for 1 ≤ i ≤ n and, moreover, N1, N2, . . . , Nn are independent processes. Note that 0 < pi < 1, for1 ≤ i ≤ n, and

∑ni=1 pi = 1. From the stated result it follows that

[(N1(t), N2(t), . . . , Nn(t))|N(t) = m] ∼ Multinomial(m, p1, p2, . . . , pn) (2.16)

for any m ∈ IN and t > 0; i.e.,

P(N1(t) = k1, N2(t) = k2, . . . , Nn(t) = kn|N(t) = m) =m!

k1!k2! . . . kn!pk11 pk2

2 . . . pknn (2.17)

for k1, k2, . . . , kn ∈ IN0 such that∑n

i=1 ki = m. In particular

(Ni(t)|N(t) = m) ∼ Bin(m, pi) (2.18)

for 1 ≤ i ≤ m.

Example 21 Suppose that the manufacturing facility of Example 19 also manufactures desktopsand that the desktop order arrivals follow a Poisson process with rate 1/h, independent of theprocess of laptop order arrivals. Compute the probability that three or more desktops are orderedbefore two laptops are ordered.

Solution: Let N = {N(t), t ≥ 0} (M = {M(t), t ≥ 0}) be the counting process of laptop (desktop)order arrivals with time in hours, so that N ∼ PP (0.5) and M ∼ PP (1) are independent processes.As a consequence, in view of (2.14), (N + M) ∼ PP (1.5).

As the probabilities in (2.17) are not a function of the time instant t, the probability that threeor more desktops are ordered before two laptops are ordered (i.e., there are three or four desktopsordered among the first four orders received) is equal to

P(M(t) ≥ 3|N(t) + M(t) = 4) = P(M(t) = 3|N(t) + M(t) = 4) + P(M(t) = 4|N(t) + M(t) = 4)

=4!

3!1!

(1

0.5 + 1

)3 (0.5

0.5 + 1

)1

+(

10.5 + 1

)4

by (2.18)

= 4881

+1681

=4881≈ 59.26%.

45

In general, the probability that m desktops are ordered before n laptops are ordered is equalto the probability that among the first (m + n − 1) orders there are at least m desktops ordered,given by

m+n−1∑

k=m

(m + n− 1

k

)(1

0.5 + 1

)k (0.5

0.5 + 1

)m+n−1−k

=m+n−1∑

k=m

(m + n− 1

k

)2k

3m+n−1

in view of (2.17). Note that an alternative derivation of this expression could be done using theprobability density and survival functions of the Erlang distribution, i.e., using formulas (2.5) and(2.6). ¥

2.2 Definition and parametrization

In this section we address the following problem.

Problem 9 Given a continuous time process X = {X(t), t ≥ 0} decide if it is a CTMC and, ifso, provide its parameterizations.

Before we define CTMCs we first give a simple example that may help to understand whatCTMCs turn out to be. This example is related to the cash register machine example consideredin Example 1. Recall that, as stated in the introduction of the chapter, CTMCs are stochasticprocesses in continuous time that have discrete state space and in addition possess the Markovproperty of the future evolution of the process being independent of its past evolution given thatits present state is known.

Example 22 A small 24 hours/day shop as a unique cash register machine. The owner of theshop collects data on the successive amounts of time the cash register is operational and underrepair. From the data he has concluded that the amounts of time (in days) the machine staysoperational without interruption have exponential distribution with parameter 0.01, the durationsof the repairs also have exponential distribution but with parameter 0.90, and all these times areindependent.

For modelling the system, let

X(t) =

{1 the cash register machine is operational at time t

0 the cash register machine is under repair at time t.

As the exponential distribution possesses the lack of memory property we can easily see that thefuture evolution of the process {X(t), t ≥ 0} is independent of its past evolution given that itspresent state (0: under repair, or 1: operational) is known. That is, {X(t), t ≥ 0} possesses theMarkov property and, thus, it is a CTMC since it has finite state space S = {0, 1}. Moreover, theconditions set forth imply that {X(t), t ≥ 0} is time-homogeneous; in this setting, this conditionmeans that the cash register machine does not age with time.

The (exponential) rates at which {X(t), t ≥ 0} makes transitions from state 0 to state 1 andfrom state 1 to state 0 are r01 = 0.90 and r10 = 0.01, respectively. Thus, a possible parametrizationof the CTMC {X(t), t ≥ 0} is through the rate matrix

R =[r00 r01

r10 r11

]=

[0 0.90

0.01 0

]

46

where rij denotes the rate at which transitions from state i to state j take place.Moreover, we can see that the (exponential) rates at which {X(t), t ≥ 0} leaves state 0 and state

1 are r0 = 0.90 and r1 = 0.01, respectively. Since the machine alternates between being operationaland being under repair, given that {X(t), t ≥ 0} makes a state transition, the transition probabilitymatrix that rules these state transitions is the embedded transition probability matrix (at transitionepochs) P = (pij)i,j=0,1, where p00 = p11 = 0 and p01 = p10 = 1. Thus, {X(t), t ≥ 0} maybe parameterized through the rate vector at which the CTMC leaves states and the embeddedtransition probability matrix that rules transitions between states when they take place,

r =[r0

r1

]=

[0.900.01

]and P =

[p00 p01

p10 p11

]=

[0 11 0

]

respectively. Note that r = R1 and that P = [diag(r)]−1R; and, in turn, R = [diag(r)]P . Alsonote the similarity of this parametrization and the parametrization of a DTMC as presented inSection 1.9.

This example is very simple because the state space has just two states, which makes the transi-tions that take place (given that the state from which the transition occurs is known) deterministic.In general, to fully explain the relation between the rate matrix R and the embedded transitionprobability matrix P that rules transitions between states, we will have to rely on properties of theexponential distribution reviewed in Section 2.1, which we will do later. ¥

A continuous time process {X(t), t ≥ 0} with discrete state space S is a CTMC if and only ifit possesses the Markov property; i.e.,

P(X(s + t) = is+t|X(u) = iu, 0 ≤ u ≤ s) = P(X(s + t) = is+t|X(s) = is) (2.19)

for any s, t ≥ 0, is+t ∈ S and iu ∈ S, for 0 ≤ u ≤ s. The statement made about the process{X(t), t ≥ 0} implies that:

· the distribution of X(s+ t) given X(u), 0 ≤ u ≤ s, is the same as the distribution of X(s+ t)given (only) X(s), for any s, t ≥ 0.

This corresponds to the way we had previously stated in words the Markov property. We willconsider only time-homogeneous CTMCs, for which X possesses the homogeneity property:

· the distribution of X(s + t) given X(s) depends on s and s + t only through their differencet = (s + t)− s, for all s, t ≥ 0.

That is,

P(X(s + t) = j|X(s) = i) = P(X(t) = j|X(0) = i) (2.20)

for any s, t ≥ 0 and i, j ∈ S.In view, of (2.19) and (2.20), a continuous time process X = {X(t), t ≥ 0} with discrete state

space S is a time-homogeneous CTMC if and only if

P(X(s + t) = j|X(s) = i,X(u) = vu, 0 ≤ u < s) = P(X(t) = j|X(0) = i) (2.21)

for any s, t ≥ 0, i, j ∈ S and vu ∈ S, for 0 ≤ u < s. From now onwards we will drop time-homogeneous when refereing to (time-homogeneous) CTMCs. Moreover we let P (t) = [pij(t)]denote the transition probability matrix of the CTMC X in t units of time; i.e.,

pij(t) = P(X(t) = j|X(0) = i) (2.22)

47

for t ≥ 0 and i, j ∈ S. In addition we let a = [ai]i∈S denote the row vector of initial probabilities,i.e.,

ai = P(X(0) = i), i ∈ S (2.23)

and a(t) = [ai(t)]i∈S denote the row vector of probabilities at time t,

ai(t) = P(X(t) = i), i ∈ S (2.24)

for t ≥ 0, so that a = a(0).{P (t), t ≥ 0} is called the transition probability function of the CTMC X. This function is

well defined provided the CTMC is stable, i.e., it makes a finite number of transitions in finitetime with probability one, independently of the initial state. In that case a and {P (t), t ≥ 0}characterize completely the distribution of the CTMC. Although non-stable CTMCs are of littleinterest in practice we provide, for the sake of completeness, an example of a CTMC that is notstable in Example 23.

From now onwards we will consider only stable CTMCs with no instantaneous states, i.e., theCTMC spends a strictly positive amount of time in each visit to each state almost surely.

We note that the transition probability matrices P (t) = [pij(t)] of a CTMC are stochasticmatrices, i.e., they have nonnegative entries and unitary row sums:

pij(t) ≥ 0 and∑

k∈S

pik(t) = 1 (2.25)

for all i, j ∈ S and t ≥ 0, and

pij(0) =

{1 j = i

0 j 6= i(2.26)

for all i ∈ S. In matrix notation, P (t) ≥ 0 and P (t)1 = 1, for all t ≥ 0, and P (0) = I. Moreover,the transition probability matrices satisfy the Chapman-Kolmogorov equations

P (s + t) = P (s)P (t), for all s, t ≥ 0. (2.27)

The Chapman-Kolmogorov equations follow from the total probability law by computing the dis-tribution of the CTMC at time s + t by conditioning on the distribution of the CTMC at time s.That is,

pik(s + t) = P(X(s + t) = k|X(0) = i)

=∑

j∈S

P(X(s + t) = k, X(s) = j|X(0) = i) by the total probability law

=∑

j∈S

P(X(s) = j|X(0) = i)P(X(s + t) = k|X(s) = j, X(0) = i)

=∑

j∈S

P(X(s) = j|X(0) = i)P(X(s + t) = k|X(s) = j) by the Markov property

=∑

j∈S

P(X(s) = j|X(0) = i)P(X(t) = k|X(0) = j) by the homegeneity property

=∑

j∈S

pij(s)pjk(t)

48

for all i, k ∈ S and s, t ≥ 0. In general, the computation of an analytic expression for P (t) isdifficult. An exception to this rule is the computation of the transition probability function ofa Poisson process, as presented in Example 24. Thus, in applications the transition probabilityfunction of a CTMC X does not play a major role in establishing that X is a CTMC.

In general, to conclude that a continuous time process X = {X(t), t ≥ 0} with discrete statespace S is a CTMC we show the following properties:

• The amount of time X stays in state i is exponentially distributed with rate ri, 0 ≤ ri < ∞,say;

• Given that the sequence of successive states visited by X, X = {Xn, n ∈ IN0} is known, theamounts of time X spends in the successive states visited are independent; and

• The sequence of successive states visited, X, is a DTMC with state space S and some tran-sition probability matrix P = [pij ]i,j∈S .

The first two properties can usually be checked simultaneously by showing that each timeX visits state i it stays in the state a random time with exponential distribution with rate ri,independently of the past history of X. The Markov property is assured by the third condition attransition epochs and by the first two conditions at non-transition epochs. By excluding ri = ∞we rule out CTMCs with instantaneous states; this does not constitute a restriction when the statespace is finite. Also note that pii = 0 for all i ∈ S, i.e., there are no self-transitions.

We will call X the embedded DTMC (at transition epochs) associated to the CTMC X and Pthe embedded transition probability matrix. If ri = 0, then once the CTMC reaches state i it staysthere forever, i.e., state i is absorbing. In that case, by convention, pii = 1 and thus pij = 0 forj 6= i; thus, state i becomes also absorbing for the embedded DTMC X.

Once we have established that X = {X(t), t ≥ 0} is a CTMC with state space S, the followingthree parameterizations of X may be used to fully characterize the distribution of X, provided thedistribution of X(0) is known:

• The vector of transition rates out of states and the embedded transition probability matrix,(r, P ), where the column vector r = [ri]i∈S and the matrix P = [pij ]i,j∈S are as defined above;

• The rate matrix of state transitions,

R = [rij ]i,j∈S = [diag(r)]P (2.28)

i.e., rij = ripij is the transition rate from state i to state j, for i, j ∈ S; and

• The (infinitesimal) generator matrix

Q = [qij ]i,j∈S = R− [diag(r)] = [diag(r)](P − I) (2.29)

i.e., qij = rij for j 6= i, and qii = −ri for i ∈ S.

Note that the most appropriate view of a CTMC X given by the rate matrix R = [rij ] is thefollowing. Each time X visits state i, independent exponential firing times Yij ∼ Exp(rij), forj ∈ S with j 6= i, that trigger transitions from state i to state j are set up. Once one of theserandom times fires, Yik say, the transition associated to the index of the firing time, from state i tostate k, is triggered. According to (2.13), which also holds for a countable number of exponential

49

random variables provided the sum of their rates is finite, the time for a transition to be triggeredhas exponential distribution with parameter ri =

∑j∈S rij , where rii = 0, and the probability that

a transition from state i to state k is triggered is

pik =rik∑j∈S rij

=rik

ri.

This agrees with (2.28) since, as P1 = 1, (2.28) leads to R1 = [diag(r)]P1 = [diag(r)]1 = r [i.e.,ri =

∑j∈S rij for i ∈ S], and, moreover, rij = ripij for i, j ∈ S. Thus, in general, given the rate

matrix R, the embedded transition probabilities are obtained in the following way

pij =

rij

riri > 0

1 ri = 0, j = i

0 ri = 0, j 6= i

. (2.30)

The generator matrix of a CTMC is closely related to the corresponding rate matrix and maybe interpreted as the derivative of the transition probability function of the CTMC at zero, i.e.,

Q = lim∆t→0+

P (t)− P (0)∆t

. (2.31)

Note that, as Q = [diag(r)](P − I), Q is also closely related to the matrix P − I. Recall thatstationary probability row vectors π of the embedded DTMC X satisfy π(P − I) = 0. Similarly,as it will be stated more precisely later, stationary probability row vectors p of X satisfy pQ = 0.

Using the Chapman-Kolmorogov equations, it may be shown that P ′(t) = QP (t) and alsoP ′(t) = P (t)Q, for all t ≥ 0; the first (second) of these set of equations are called the backward(forward) Kolmogorov equations. The solution of the Kolmogorov equations is

P (t) = eQt =∞∑

k=0

(Qt)k

k!= I + Qt +

(Qt)2

2!+

(Qt)3

3!+ . . . (2.32)

for t ≥ 0. When the rates out of states are bounded, a computational alternative to compute P (t)is based on a technique called uniformization, which we will study in Section 2.3.

Example 23 An example of a CTMC that is not stable is the CTMC X = {X(t), t ≥ 0} withstate space IN0, such that: the rate at which the CTMC leaves state i is ri = (i + 1)2 and adeterministic transition to state i + 1 occurs when X leaves state i, i.e., the embedded transitionprobabilities (at transition epochs) are pi,i+1 = 1 and pij = 0 for j 6= i + 1. Thus, the vector oftransitions rates out of states of X and the embedded transition probability matrix are

r =

14916. . .

and P =

0 1 0 0 0 . . .0 0 1 0 0 . . .0 0 0 1 0 . . .0 0 0 0 1 . . .

. . . . . . . . . . . . . . . . . .

.

Now, if we suppose that P (X(0) = 0) = 1 and define S0 = 0 and

Si = inf{t ≥ Si−1 : X(t) = i}

50

recursively for i ∈ IN , we can conclude using the Markov property of X that Yi = Si−Si−1, i ∈ IN ,are independent random variables and Yi ∼ Exp(i2). Thus, if we let S =

∑∞i=1 Yi,

E[S] = E

[ ∞∑

i=1

Yi

]=

∞∑

i=1

E[Yi] =∞∑

i=1

1i2

=π2

6< ∞.

Since S has finite expected value, then S is finite almost surely; i.e., P(S < ∞) = 1. But then,since S is finite if and only if the CTMC X makes an infinte number of transitions in finite time,this implies that X is not stable. ¥

In the rest of the chapter we will consider stable CTMCs with no instantaneous states.

Example 24 Let N = {N(t), t ≥ 0} be a Poisson process with rate λ, N ∼ PP (λ), so that N(t)represents the number of events observed in the interval [0, t]. Since a PP (λ) is a counting processsuch that N(0) = 0 and the inter-event times are independent and identically distributed randomvariables with exponential distribution with parameter λ, it is also a CTMC with:

· state space IN0;

· the rate at which the CTMC leaves state i is λ, for all i ∈ IN0; and

· a deterministic transition to state i + 1 occurs when X leaves state i, for all i ∈ IN0.

Thus, the vector of transitions rates out of states of N and the embedded transition probabilitymatrix are

r =

λλλλ. . .

and P =

0 1 0 0 0 . . .0 0 1 0 0 . . .0 0 0 1 0 . . .0 0 0 0 1 . . .. . . . . . . . . . . . . . . . . .

so that P is the same as in Example 23. Note also that the rate and generator matrix of N are

R =

0 λ 0 0 0 . . .0 0 λ 0 0 . . .0 0 0 λ 0 . . .0 0 0 0 λ . . .

. . . . . . . . . . . . . . . . . .

and Q =

−λ λ 0 0 0 . . .0 −λ λ 0 0 . . .0 0 −λ λ 0 . . .0 0 0 −λ λ . . .

. . . . . . . . . . . . . . . . . .

.

A second definition of a Poisson process with rate λ is thus as being a CTMC N = {N(t), t ≥ 0}with state space IN0 and rate matrix R as given in the previous equation, such that N(0) = 0.

Now, if we define S0 = 0 and

Si = inf{t ≥ Si−1 : X(t) = i}recursively for i ∈ IN , then Xi = Si − Si−1, i ∈ IN , are independent random variables withexponential distribution with parameter λ. The random variables Xi denote the inter-event timesbetween the occurrence of the (i− 1)-th and ith events and Si denotes the time of the occurrenceof the i-th event of the Poisson process N . Now, in view of (2.6) or (2.8),

P(N(t) ≤ n) = P(Sn+1 > t) = P

(n+1∑

i=1

Xi > t

)=

n∑

k=0

e−λt (λt)k

k!

51

for all n ∈ IN0 and t ≥ 0. Proceeding similarly, we could conclude, in addition, that

P(N(s + t) = j|N(s) = i) =

{e−λt (λt)j−i

(j−i)! j ≥ i

0 j < i(2.33)

for all s, t ≥ 0.As P(N(s+t) = j|N(s) = i) = P(N(t) = j−i|N(0) = 0), for all states i and j and times s and t,

the Poisson process is space-homogeneous in addition to being time-homogeneous. The space-time-homogeneity of the Poisson process is usually phrased by saying that the Poisson process possessesstationary increments, which stands for the distribution of the increments N(t + s) −N(s) of thePoisson process depending only on the size of the interval (s, t], but not on the specific interval oftime, which in view of (2.33) is true since

N(s + t)−N(s) ∼ P (λt), for all s, t ≥ 0.

Moreover, from the Markov property of the Poisson process it follows that N(s + t) −N(s) isindependent of N(s) for all s, t ≥ 0; i.e., the Poisson process has independent increments. In fact,a third definition for N = {N(t), t ≥ 0} being a Poisson process with rate λ is:

• N(0) = 0;

• N has independent increments; and

• N(s + t)−N(s) ∼ P (λt), for all s, t ≥ 0.

You can check on how the properties of the Poisson process presented in Section 2.1 follow fromthis alternative definition by using properties related to the Poisson and binomial distributions. ¥

Example 25 A machine produces components according to a Poisson process {N(t), t ≥ 0} withrate λ. An employee packs the components by the order in which they are produced. The packingtimes are independent random variables with exponential distribution with rate µ, independent ofproduction times. Show that if we let

X(t) = number of produced components that have not been packed at time t

for t ≥ 0, then X = {X(t), t ≥ 0} is a CTMC and obtain its rate matrix.

Solution: It is clear that X has state space IN0. When X is in state 0, i.e., there are no itemsto be packed, the next transition (from state 0 to state 1) occurs when a new item is produced,which occurs after a random time with exponential distribution with parameter λ. Similarly, whenX is in state i, i ∈ IN , (using the lack of memory of the exponential distribution) either one of twoindependent exponential random variables may trigger a transition: an Exp(λ) random variablecorresponding to the production time of a component that triggers a transition to state i + 1,and an Exp(µ) random variable corresponding to the packing time of a component that triggers atransition to state i− 1.

As all production and packing times are independent, then X is a CTMC with state space IN0

and rate matrix

R =

0 λ 0 0 0 . . .µ 0 λ 0 0 . . .0 µ 0 λ 0 . . .0 0 µ 0 λ . . .

. . . . . . . . . . . . . . . . . .

.

52

In addition to the transition rates from state i to state (i + 1), i ∈ IN0, characteristic of the PP (λ)associated to the production of components, this rate matrix exhibits positive transition rates fromstate i to state (i− 1), i ∈ IN , associated to the packing component completions. ¥

We note that the transition rates rij of the CTMC X of Example 25 are an instance of thefollowing generic form of transition rates

rij =

λi j = i + 1µi j = i− 10 j 6= i− 1, i + 1

. (2.34)

CTMCs with transition rates of the form (2.34) are called birth-death processes because if the stateis interpreted as a population size, then: a transition from state i to state (i + 1) corresponds to abirth and a transition from state i to state (i− 1) to a death.

Birth-death processes enjoy many pleasant properties and have many applications aside fromthe study of populations. In the course we will emphasize their applications in queueing networks.For birth-death processes with transition rates of the form (2.34) it is common to specify theprocess just by saying that it is a birth-death process with birth rates {λi, i ∈ IN0} and death rates{µi, i ∈ IN}.

Thus, e.g., the CTMC X of Example 25 is a birth death process on IN0 with constant birth rateλ and constant death rate µ. In fact, X corresponds to the number of customers in an M/M/1queueing system (a system with one server, exponential service times and arrivals following aPoisson process) which we have been studying in class and that will be presented later in thesenotes in more detail.

We next show one example of the direct computation of transition probability matrices ofCTMCs using (2.32).

Example 26 We compute the transition probability matrix of the CTMC X of Example 22 in 3and 10 units of time, P (3) = e3Q and P (10) = e10Q, with

Q = R− diag(r) =[

0 0.900.01 0

]−

[0.90 00 0.01

]=

[−0.90 0.900.01 −0.01

].

The computation of P (3) = e3Q and P (10) = e10Q with Mathematica gives

P (3) =[0.075492 0.9245080.010272 0.989728

], P (10) =

[0.011099 0.9889010.010988 0.989012

].

Note that for the closely related DTMC of Example 4 the transition probability matrices in 3 and10 steps are

P 3 ≈[0.011710 0.9882900.010981 0.989019

], P 10 ≈

[0.010989 0.9890110.010989 0.989011

].

Observe that the differences between the transition probabilities for the continuous and discretetime models are non-negligible for instants separated by 3 units of time but basically negligiblefor instants separated by 10 units. We may anticipate from this that the distribution of the stateafter 10 units of time will be very close to the steady-state distribution (both in continuous and indiscrete time). ¥

53

2.3 Uniformization

Note that though the CTMCs of examples 23 and 24 have the same state space and embeddedtransition probability matrix P they have very different properties. This is due to the fact that thetransition rates out of states are very different in the two examples.

In fact, different transitions rates out of states is what makes the study of CTMCs (a little)more complex than the study of DTMCs. For suppose that all rates out of states of the CTMCX = {X(t), t ≥ 0} with state space S are equal to some value q > 0; and let X = {Xn, n ≥ 0} bethe embedded DTMC associated to X, P be the corresponding embedded transition probabilitymatrix, and Nq(t) denote the number of state transition made by X on the time interval [0, t].

Then, Nq = {Nq(t), t ≥ 0} ∼ PP (q), X is a DTMC with transition probability matrix P , and,moreover, Nq and X are independent. Thus,

pij(t) = P(X(t) = j|X(0) = i)

=∞∑

n=0

P(Nq(t) = n)P(X(t) = j|X(0) = i,Nq(t) = n) by the total probability law

=∞∑

n=0

P(Nq(t) = n)P(Xn = j|X0 = i) since X is independent of Nq

=∞∑

n=0

e−qt (qt)n

n!p(n)ij since Nq ∼ PP (q) ⇒ Nq(t) ∼ P (qt)

for any i, j ∈ S and t ≥ 0. In matrix form,

P (t) =∞∑

n=0

e−qt (qt)k

k!Pn, t ≥ 0. (2.35)

This relation forms the basis for the derivation of the technique called uniformization whichis very important for the computation of transition probability matrices of CTMCs with boundedrates out of states. To derive the uniformization technique we just need to add another simple idea:

Adding (fictitious) self-transitions to a CTMC does not change the distribution of the CTMC.

A rigorous justification of this idea now follows.If the vector r of rates out of states of the CTMC X with rate matrix R is bounded by some

rate q > 0, i.e., |ri| ≤ q for all i ∈ S, we say that X is uniformizable. If X is uniformizable, then Xhas the same distribution as the CTMC Y = {Y (t), t ≥ 0} with rate matrix R = [rij ]ij∈S

rij =

{rij j 6= i

q − ri j = i(2.36)

since X and Y have the same generator matrix

Q = R− diag(R1) = R− diag(R1) (2.37)

and taking into account (2.32).Note that the CTMC Y is obtained from X by adding self-transitions with rate q − ri when

the process is in state i ∈ S. That is, the CTMC is uniformized, in the sense that in all statestransitions are initiated with the same rate q (although some may now be self-transitions).

54

We will let X = {Xn, n ≥ 0} denote the embedded DTMC associated to Y and will callit embedded uniformized DTMC associated to X with uniformization rate q (we will drop theuniformization rate, whenever possible, to ease the notation). Moreover we will let P denote thetransition probability matrix of X an will call it embedded uniformized transition probability matrixof X (at transition epochs). Taking into account (2.30), it follows that

pij =

{rij

q j 6= i

1− riq j = i

(2.38)

or, in matrix form,

P = I +Q

q. (2.39)

We may now finally conclude the derivation of the uniformization technique. As

pij(t) = P(X(t) = j|X(0) = i)= P(Y (t) = j|Y (0) = i) by (2.32) + (2.37)

=∞∑

n=0

e−qt (qt)n

n!p(n)ij by (2.35)

for any i, j ∈ S and t ≥ 0, where p(n)ij = [Pn]ij . In matrix form,

P (t) =∞∑

n=0

e−qt (qt)n

n!Pn, t ≥ 0. (2.40)

We call (2.40) the uniformization formula (for the transition probability function of X) associatedwith the uniformization rate q.

Note that the infinite sum (2.40) has to be truncated for effects of computation. If we denoteby P (k)(t) the approximation of P (t) using the terms of (2.40) corresponding to n = 0, 1, 2, . . . , k,i.e.,

P (k)(t) =k∑

n=0

e−qt (qt)n

n!Pn, t ≥ 0

then we will have

|p(k)ij (t)− pij(t)| ≤

∞∑

n=k+1

e−qt (qt)n

n!= P(Nq(t) > k)

for all i, j ∈ S. Thus, to assure a precision of at least ε we may choose the smallest k such that

P(Nq(t) > k) =∞∑

n=k+1

e−qt (qt)n

n!≤ ε

i.e., choose k as being the quantile of probability 1− ε of the Poisson distribution with mean qt.For example, if we choose the precision ε = 0.5 × 10−6 for the CTMC X of Example 26 with

q = 0.90, we obtain the approximations given there for P (3) and P (10) for the values k = 14 andk = 27, respectively.

55

Example 27 Consider the modification of Example 25 where there can be at most 2 componentswaiting to be packed, so that the machine is turned off when there the number of components notyet packed reaches 3 (1 being packed and other 2 waiting for starting being packed) and turned onagain when it gets below 3.

In this situation, if we let again X(t) denote the number of produced components that have notbeen packed at time t, for t ≥ 0, then X = {X(t), t ≥ 0} is a CTMC with state space {0, 1, 2, 3}and rate matrix

R =

0 λ 0 0µ 0 λ 00 µ 0 λ0 0 µ 0

.

For λ = µ = 2 compute the transition probability matrix of X in 1.2 and 7 units of time withaccuracy at least 0.5%.

Solution: As

R =

0 2 0 02 0 2 00 2 0 20 0 2 0

we may choose q = 4, in which case the embedded uniformized transition probability matrix is

P =

1/2 1/2 0 01/2 0 1/2 00 1/2 0 1/20 0 1/2 1/2

.

Moreover, the quantiles of probability 0.995 of the Poisson distributions with parameters 4.8 =4× 1.2 and 28 = 4× 7 are 11 and 43, respectively, and

P (1.2) ≈

0.356 0.290 0.204 0.1460.290 0.269 0.233 0.2040.204 0.233 0.269 0.2900.146 0.204 0.290 0.356

, P (7) ≈

0.249 0.249 0.249 0.2490.249 0.249 0.249 0.2490.249 0.249 0.249 0.2490.249 0.249 0.249 0.249

are approximations of the transition probability matrices of X in 1.2 and 7 units of time withaccuracy at least 0.5%. ¥

Uniformization provides an alternative to the computation of P (t) = eQt when the rates outof states are bounded. In general, uniformization has better computational properties than thedirect computation of eQt; in addition it provides a probabilistic interpretation of the transitionprobability matrix of a CTMC. Note that all CTMCs with finite state space have bounded ratesout of states, so that uniformization can be used for all finite state CTMCs, and also for most ofthe infinite-state CTMCs of practical interest.

Uniformization is usually used as a computational technique and not as an analytical technique,but it can also used as an analytical technique provided we are able to derive an analytical formulafor the embedded uniformized transition probability matrix P as shown in the next example.

Example 28 Consider the general version of the Example 22, where

X(t) =

{1 the machine is operational at time t

0 the machine is under repair at time t

56

now with generator matrix

Q =[−λ λ

µ −µ

].

The CTMC X is uniformizable, and if we choose for the uniformization rate λ + µ, then theassociated embedded uniformized transition probability matrix is

P = I +Q

λ + µ=

[µ

λ+µλ

λ+µµ

λ+µλ

λ+µ

].

Moreover, it follows that Pn = P for all n ∈ IN . Therefore, the uniformization formula (2.40) gives

P (t) =∞∑

n=0

e−(λ+µ)t [(λ + µ)t]n

n!Pn by (2.40) since q = λ + µ

= e−(λ+µ)tI + (1− e−(λ+µ)t)P since Pn = P, for n ∈ IN

= P − e−(λ+µ)t(P − I) rearranging terms

= P − e−(λ+µ)t Q

λ + µsince (P − I) = Q/q

=

[µ

λ+µλ

λ+µµ

λ+µλ

λ+µ

]+ e−(λ+µ)t

[λ

λ+µ − λλ+µ

− µλ+µ

µλ+µ

].

Thus, in particular,

limt→+∞P (t) = P =

[µ

λ+µλ

λ+µµ

λ+µλ

λ+µ

]

and the rate of convergence of P (t) to its limit distribution is exponential. ¥

2.4 Transient quantities

In this section we address the following problem.

Problem 10 Given a CTMC compute probabilities associated with the state of the CTMC at afinite number of instants.

Once we have a way to compute the transition probability matrices P (t), t ≥ 0, of a CTMCX, we can compute joint probabilities of the state of the CTMC at a finite number of instants0 ≤ t1 < t2 < . . . , < tn by the formula

P(X(t1) = i1, X(t2) = i2, . . . , X(tn) = in) =∑

i0∈S

ai0

n∏

j=1

pij−1ij (tj − tj−1) (2.41)

with t0 = 0, for all i1, i2, . . . , in ∈ S. This formula is analogous to (1.9) and also follows using theMarkov property and the composition rule, since

P(X(t1) = i1, X(t2) = i2, . . . , X(tn) = in)

=∑

i0∈S

P(X(0) = i0, X(t1) = i1, X(t2) = i2, . . . , X(tn) = in) by the total probability law

=∑

i0∈S

P(X(0) = i0)n∏

j=1

P(X(tj) = ij |X(tk) = ik, 0 ≤ k ≤ j − 1) by the composition rule

57

=∑

i0∈S

P(X(0) = i0)n∏

j=1

P(X(tj) = ij |X(tj−1) = ij−1) by the Markov property

=∑

i0∈S

ai0

n∏

j=1

pij−1ij (tj − tj−1) by the homogeneity property.

Moreover, (2.41) may be used to compute quantities (like expected values, variances, covariances,and others) associated with the random vector (X(t1), X(t2), . . . , X(tn)). Thus, the all programthat has been done for the transient analysis of DTMCs may be translated directly for the transientanalysis of CTMCs with real times replacing integer times and P (t) = eQt replacing P (n) = Pn.

Example 29 Consider the CTMC X of Example 27, for which X(t) denotes the number of pro-duced components that have not been packed by time t, for t ≥ 0, and assume that initially thereno components whose packing has not yet been finished.

Compute the expected number of components that have not been packed at time 1.2 and theprobability that there is 1 component that has not been packed at time 1.2 and 2 components thathave not been packed at time 8.2.

Solution: From Example 27, the transition probability matrices of X in 1.2 and 8.2 units of timeare approximately

P (1.2) ≈

0.356 0.290 0.204 0.1460.290 0.269 0.233 0.2040.204 0.233 0.269 0.2900.146 0.204 0.290 0.356

, P (7) ≈

0.249 0.249 0.249 0.2490.249 0.249 0.249 0.2490.249 0.249 0.249 0.2490.249 0.249 0.249 0.249

and, in addition, the initial distribution of X is a =[1 0 0 0

]. Thus, we have

E[X(1.2)] =3∑

k=0

kP(X(1.2) = k)

=3∑

k=0

kP(X(1.2) = k|X(0) = 0) since X(0) = 0

=3∑

k=0

kp0k(1.2) ≈ 0× 0.356 + 1× 0.290 + 2× 0.204 + 3× 0.146 = 1.136

so that the the expected number of components that have not been packed at time 1.2 is 1.136.Moreover, the the probability that there is 1 components that has not been packed at time 1.2 and2 component that have not been packed at time 8.2 is

P(X(1.2) = 1, X(8.2) = 2) = P(X(1.2) = 1, X(8.2) = 2|X(0) = 0) since X(0) = 0= P(X(1.2) = 1|X(0) = 0)P(X(8.2) = 2|X(1.2) = 1) by the Markov property of X

= P(X(1.2) = 1|X(0) = 0)P(X(7) = 2|X(0) = 1) by the homogeneity property of X

= p01(1.2)× p12(7)≈ 0.290× 0.249 = 0.072. ¥

58

2.5 Expected accumulated cost/rewards

In this section we will show how expected accumulated cost/rewards associated with finite stateCTMCs may be derived. We consider the situation where costs/rewards may be incurred bothcontinuously at a rate dependent on the state of the CTMC and discretely when transitions between(different) states of the CTMC take place.

That is, for the CTMC X with (finite) state space S, transition probability function {P (t), t ≥0} and embedded uniformized transition probability matrix P associated to the uniformization rateq, the cost C(t) incurred until time t is of the form

C(t) =∫ t

0cX(s) ds +

Nq(t)∑

k=1

c(Xk−1,Xk) (2.42)

where cj is the cost rate incurred continuously when the CTMC is in state j and c(i,j) is the costincurred by a transition from state i to state j, where c(i,i) = 0 for all i ∈ S, and {Nq(t), t ≥ 0} isa Poisson process with rate q independent of the embedded uniformized DTMC X. The formulawe will derive for the expected value of C(t) is also valid if instead the costs incurred at transitionsepochs are independent random variables such that the expected cost associated with a transitionfrom state i to state j is c(i,j). In the section we address the following problem.

Problem 11 For accumulated cost/rewards of the form (2.42), compute the expected accumulatedcost/reward incurred/obtained until a finite time t.

As introduced in Section 2.2, we let a = [ai]i∈S denote the initial probability row vector of Xand X; i.e.,

ai = P(X(0) = i) = P(X0 = i), i ∈ S.

and c = [cj ]j∈S denote the column vector of cost rates incurred continuously in the states of X.Moreover, as in Section 1.4, we let C = [cij ]i,j∈S be the matrix of costs associated with statetransitions in the CTMC X (or in X). Then, we have the following result. The proof of the resultis long; however it is enlightening because it incorporates the majority of the results we have derivedso far for the transient case in an effective way.

Theorem 2 With the setting described for the finite state CTMC X and C(t) given by (2.42), wehave:

E[C(t)] =∞∑

n=0

P(Nq(t) > n)[1q

aPnc + aPn(P • C)1]

(2.43)

= a∞∑

n=0

P(Nq(t) > n)Pn[c/q + (P • C)1

](2.44)

for all t > 0. Moreover, if we let

α =1q

maxi∈S

|ci|+ maxi,j∈S

|c(i,j)|

then, for ε > 0,∣∣∣∣∣E[C(t)]−

k∑

n=0

P(Nq(t) > n)[1q

aPnc + aPn(P • C)1]∣∣∣∣∣ ≤ ε

59

provided thatk∑

n=0

P(Nq(t) > n) ≥ qt− ε

α. (2.45)

Proof: We will prove (2.43); the other result of the theorem follows from the properties of absolutevalues and the fact that, since Nq(t) ∼ P (qt),

qt = E[Nq(t)] =∞∑

n=0

P(Nq(t) > n)

for all t > 0. We start by computing E[∑Nq(t)

k=1 c(Xk−1,Xk)

]

E

Nq(t)∑

k=1

c(Xk−1,Xk)

= E

E

Nq(t)∑

k=1

c(Xk−1,Xk)

∣∣∣∣∣∣Nq(t)

since E[Z] = E[E[Z|W ]]

=∞∑

m=0

P(Nq(t) = m)E

Nq(t)∑

k=1

c(Xk−1,Xk)

∣∣∣∣∣∣Nq(t) = m

=∞∑

m=0

P(Nq(t) = m)E

[m∑

k=1

c(Xk−1,Xk)

]

=∞∑

m=0

P(Nq(t) = m)m∑

k=1

aP k−1(P • C)1 by (1.15)

=∞∑

m=0

P(Nq(t) = m)m−1∑

n=0

aPn(P • C)1 making n = k − 1

=∞∑

n=0

∞∑

m=n+1

P(Nq(t) = m)aPn(P • C)1 interchanging m and n

=∞∑

n=0

P(Nq(t) ≥ n + 1)aPn(P • C)1

=∞∑

n=0

P(Nq(t) > n)aPn(P • C)1.

Thus,

E

Nq(t)∑

k=1

c(Xk−1,Xk)

=

∞∑

n=0

P(Nq(t) > n)aPn(P • C)1. (2.46)

If we note that∫ t

0cX(s) ds =

∫ t

0

∑

k∈S

ck 1{X(s)=k} ds =∑

k∈S

ck

∫ t

01{X(s)=k} ds

60

we conclude that

E[∫ t

0cX(s) ds

]= E

[∑

k∈S

ck

∫ t

01{X(s)=k} ds

]=

∑

k∈S

ckE[∫ t

01{X(s)=k} ds

]

=∑

k∈S

ck

∫ t

0E

[1{X(s)=k}

]ds =

∑

k∈S

ck

∫ t

0P(X(s) = k) ds.

Thus, we have

E[∫ t

0cX(s) ds

]=

∑

k∈S

ck

∫ t

0P(X(s) = k) ds by the previous equation

=∑

k∈S

ck

∫ t

0

∞∑

n=0

e−qs (qs)n

n!

[aPn

]k

ds by (2.40)

=1q

∞∑

n=0

∑

k∈S

ck

∫ t

0

e−qsq(qs)n

n!ds

[aPn

]k

rearranging terms

=1q

∞∑

n=0

∑

k∈S

ck

∫ t

0fSn+1)(s) ds

[aPn

]k

by(2.7) with Sn+1 ∼ Erlang(n + 1, q)

=1q

∞∑

n=0

∑

k∈S

ckP(Sn+1 ≤ t)[aPn

]k

=1q

∞∑

n=0

∑

k∈S

ckP(Nq(t) ≥ n + 1)[aPn

]k

by (2.6) + (2.8)

=1q

∞∑

n=0

P(Nq(t) > n)∑

k∈S

ck

[aPn

]k

rearranging terms

=1q

∞∑

n=0

P(Nq(t) > n)aPnc rearranging terms

and, thus,

E[∫ t

0cX(s) ds

]=

1q

∞∑

n=0

P(Nq(t) > n)aPnc. (2.47)

Now, (2.43) follows from (2.46) and (2.47). ¥Note that, similarly to (2.47), we could also conclude that

E[∫ t

01{X(s)=j} ds

∣∣∣∣X(0) = i

]=

[1q

∞∑

n=0

P(Nq(t) > n)Pn

]

ij

(2.48)

for all i, j ∈ S. That is, the expected time spent in state j on the interval [0, t] starting from statei the the entry in row i and column j of the matrix 1

q

∑∞n=0 P(Nq(t) > n)Pn.

Example 30 Let X(t) represent the state of a continuous-time (1, 3) inventory system at time th,where the state is the number of items in stock. An (s, S) inventory system is such that replen-ishment orders aimed at pushing the inventory level to the restocking level S are instantaneouslymade when the net inventory level drops strictly below the basestock level s.

61

Suppose that unitary orders from customers arrive according to a Poisson process with rate 0.1/hand that each replenishment order is satisfied after a random time (h) with exponential distributionwith parameter 0.5, independently of other orders. In addition assume that the inventory level isinitially at its maximum.

Compute the expected revenue obtained in the first 10 h if the hourly cost rate per item is $0.1,the cost of a replenishment order is $5, and the profit of selling each item is $40.

Solution: As explained in the solution to Problem 2 of Homework # 5, X = {X(t), t ≥ 0} is aCTMC with state space S = {0, 1, 2, 3}, generator matrix Q, and embedded uniformized transitionprobability matrix P associated with the uniformization rate 0.5,

Q =

−0.5 0 0 0.50.1 −0.1 0 00 0.1 −0.1 00 0 0.1 −0.1

, P = I +

Q

0.5=

0 0 0 10.2 0.8 0 00 0.2 0.8 00 0 0.2 0.8

.

and the initial probability row vector is a =[0 0 0 1

]since the the inventory level is initially at

its maximum. Moreover, if the hourly cost rate per item is $0.1, the cost of a replenishment order is$5, and the profit of selling each item is $40, the column vector of cost rates incurred continuouslyin the states of X, c, and the matrix of costs associated with state transitions in the CTMC X, C,are

c =

00.10.20.3

, C =

0 0 0 5−40 0 0 00 −40 0 00 0 −40 0

.

Thus, in view of (2.43), the expected revenue (in dollars) obtained in the first 10 h is:

E[−C(10)] ≈ −14∑

n=0

P(N0.5(10) > n)[

10.5

aPnc + aPn(P • C)1]

= 36.78

with an error of at most half a cent, where N0.5(10) ∼ P (5). ¥

2.6 First passage times

In this section we will address the following general problem.

Problem 12 Given a uniformizable CTMC, compute the distributions and expected values of thetimes to reach states of a strict subset of the state space. ¥

Suppose that X = {X(t), t ≥ 0} is a uniformizable CTMC with state space S and generatormatrix Q and let X = {Xn, n ≥ 0} denote the corresponding embedded uniformized DTMC withuniformization rate q, thus having state space S and transition probability matrix P = I + Q/q.

Moreover let {Nq(t), t ≥ 0} denote a Poisson process with rate q independent of the embeddeduniformized DTMC X, and let {Yn, n ≥ 1} be the sequence of inter-event times of {Nq(t), t ≥ 0},so that

{Yn, n ≥ 1} iid∼ Exp(q).

Thus in particular, E[Yn] = 1/q, for all n ∈ IN .

62

Consider a strict subset of states, A, such that starting from any state in the complement of A,A = S \A, with probability one the CTMC X will make a visit to a state in A at some future time.This is equivalent to stating that starting from any state in the complement of A with probabilityone the embedded uniformized DTMC X will make visit to a state in A at some future instant. If,as in Section 1.8 we order the states of X and X so that the states in A appear first, the matrix Pmay be written in block matrix as

P =[PAA PAA

PAA PAA

]

where, e.g., PAA = [pij ]i,j∈A.For i ∈ A, we let TiA denote the amount of time the CTMC X needs to make the first visit to

a state in A starting from state i, and NiA denote the number of steps the embedded uniformizedDTMC X needs to make a visit to a state in A starting from state i. That is, for i ∈ A,

TiA = [inf{t ≥ 0 : X(t) ∈ A}|X(0) = i]

NiA =[inf{n ≥ 0 : Xn ∈ A}|Xn = i

].

As TiAd=

∑NiAn=1 Yn, using Wald’s equation, we have

E[TiA] = E

NiA∑

n=1

Yn

= E[NiA]E[Y1] =

1qE[NiA].

Moreover, the expected value of the first passage time of X into set A starting from state i,E[NiA], may be computed using the results of Section 1.8. Namely, if we let miA = E[TiA] andmiA = E[NiA], then the column vector m.A = [E[NiA]]i∈A satisfies m.A = [I − PAA]−11 and, thus,the column vector m.A = [E[TiA]]i∈A is given by

m.A =1q

m.A =1q[I − PAA]−11. (2.49)

The distribution function of TiA, i ∈ A, may also be expressed as a function of the distributionfunction of NiA, since

P(TiA ≤ t) = P

NiA∑

n=1

Yn ≤ t

= P

(NiA ≤ Nq(t)

).

for i ∈ A and t ≥ 0. Now, as Nq(t) and NiA are independent, we have

P(TiA ≤ t) =∞∑

n=1

P(Nq(t) = n)P(NiA ≤ n) =∞∑

m=1

P(Nq(t) ≥ n)P(NiA = n)

and, using the fact that from Section 1.8

[P(NiA = n)

]i∈A

= Pn−1AA

(I − PAA)1,[P(NiA ≤ n)

]i∈A

=n∑

m=1

Pm−1AA

(I − PAA)1

63

for n ∈ IN , we get

[P(TiA ≤ t)]i∈A =∞∑

n=1

P(Nq(t) = n)n∑

m=1

Pm−1AA

(I − PAA)1 (2.50)

=∞∑

n=0

P(Nq(t) > n)PnAA(I − PAA)1. (2.51)

Thus, either one of the two previous formulas may be used to compute the vector [P(TiA ≤ t)]i∈A.To achieve precision ε using (2.50) ((2.51)) we just need to use the terms for n = 1, . . . , k, wherek is the quantile of probability 1 − ε of the Poisson distribution with parameter qt (the smallestvalue satisfying (2.45)).

Example 31 Consider the CTMC X of the (1, 3) inventory system described in Example 30.Compute with precision ε = 0.5 × 10−4 the expected time for a replenishment order to be

satisfied starting from 0, 1, and 2 items in stock. In addition compute the probabilities that areplenishment order is satisfied within 10 h starting from 0, 1, and 2 items in stock.

Solution: X = {X(t), t ≥ 0} is a CTMC with state space S = {0, 1, 2, 3}, generator matrix Q,and embedded uniformized transition probability matrix P associated with the uniformization rate0.5,

Q =

−0.5 0 0 0.50.1 −0.1 0 00 0.1 −0.1 00 0 0.1 −0.1

, P = I +

Q

0.5=

0 0 0 10.2 0.8 0 00 0.2 0.8 00 0 0.2 0.8

.

A replenishment order is satisfied when a transition from state 0 to state 3 in X takes place or,alternatively, when X enters state 3. Thus, if we let A = {3} and A = {0, 1, 2}

Ti3 = TiA = [inf{t ≥ 0 : X(t) = 3}|X(0) = i]

for i = 0, 1, 2, we have to compute E[Ti3] and P(Ti3 ≤ 10), for i = 0, 1, 2, with precision ε =0.5× 10−4. Taking into account (2.49)-(2.50), we have

m03

m13

m23

=

E[T03]E[T13]E[T23]

=

10.5

(I − PAA)−11 = 2

1 0 0−0.2 0.2 0

0 −0.2 0.2

−1

111

=

21222

and P(T03 ≤ 10)P(T13 ≤ 10)P(T23 ≤ 10)

≈

16∑

n=1

e−5 5n

n!

n∑

m=1

0 0 00.2 0.8 00 0.2 0.8

m−1

100

=

0.99320.54180.1948

.

since the quantile of probability 0.99995 of the Poisson distribution with parameter 5 = 0.5× 10 is16. ¥

Note that first passage time probabilities into set A in t or less units of time may also becomputed as regular probabilities of the state of a modified CTMC at time t belonging to A;namely, the modified CTMC is obtained from the original by making the states of A absorbing(i.e., if i ∈ A, then rij = 0 for all j). This is the approach that should be followed to compute first

64

passage time probabilities using MAXIM. In the previous example, this would lead to the followingcomputation

P(T03 ≤ 10)P(T13 ≤ 10)P(T23 ≤ 10)P(T33 ≤ 10)

= exp

10

−0.5 0 0 0.50.1 −0.1 0 00 0.1 −0.1 00 0 0 0

0001

≈

0.99320.54180.1948

1

where T33 = [inf{t ≥ 0 : X(t) = 3}|X(0) = 3] = 0.

2.7 Long-run and limit related quantities

In this section we will derive long-run and limit quantities associated to a CTMC, like long-runproportion of time spent in each state, the limit probability of being in a state, and the long runcost per unit time of some (cost) function associated to the sojourn times in the states and thetransitions between different states made by the CTMC.

In the section we consider X = {X(t), t ≥ 0} to be a CTMC with state space S, rate matrixR, vector of rates out of states r, generator matrix Q, and embedded transition probability matrixP . We let X = {Xn, n ≥ 0} denote the embedded DTMC at transition epochs. Moreover, if X isuniformizable, we let X = {Xn, n ≥ 0} denote a corresponding embedded uniformized DTMC andlet q denote the associated uniformization rate.

We first not that X is irreducible (i.e., for all states i and j, the probability of making a visit tostate j starting from state i is positive) if and only if the embedded DTMC X is irreducible. If X isuniformizable, then this happens if and only if the embedded uniformized DTMC X is irreducible.Aperiodicity is not an issue for CTMCs as if X starts in state i, then there is a positive probabilitythat X stays always in state i on the time interval [0, t] for any t > 0.

We let p = [pi]i∈S , π = [πi]i∈S and π = [πi]i∈S denote probability row vectors (i.e., nonnegativevectors such that p1 = π1 = π1 = 1). Note that π (π) is a stationary vector of X (X) if and onlyif π = πP (π = πP )

Note that if the CTMC X is uniformizable, then the long-run proportion of time X spends ina state equals the long-run fraction of instants at which the embedded uniformized DTMC X is inthe same state. Moreover, if π is a stationary vector of the embedded uniformized DTMC X, thenwe have

π = πP ⇐⇒ π = π

(I +

Q

q

)⇐⇒ π = π +

1qπQ ⇐⇒ πQ = 0

where 0 is a row vector of zeros. This motivates the following important auxiliary result.

Lemma 2 (Stationary distribution of an irreducible DTMC)

If the CTMC X is irreducible, then there is at most one probability row vector that is solutionof the system of linear equations

pQ = 0 (2.52)

i.e., ∑

i6=j

pi rij = pj rj , j ∈ S (2.53)

and ∑

j∈S

pj = 1. (2.54)

65

In that case p is called the stationary probability row vector (or stationary distribution) of X.If the irreducible CTMC X has finite state space then the stationary distribution exists (and isunique). ¥

The reason to call a probability row vector solution of (2.52) stationary distribution of X is thefact that: if the initial distribution of X is p, then X is a stationary CTMC, i.e.,

(X(t1), X(t2), . . . , X(tn)) d= (X(t1 + s), X(t2 + s), . . . , X(tn + s))

for all n ∈ IN and s, t1, t2, . . . , tn ≥ 0; in particular, the distribution of X(t) is the same for allvalues of t.

The equations (2.53) are known as the (global) balance or steady-state equations because if∑

i 6=j

pi rij = pj rj

then, under the probability distribution p the input rate into state j (∑

i6=j pirij) equals the outputrate from state j (pjrj). In general, we can test if a probability row vector is a stationary probabilityrow vector by checking if it satisfies (2.52) or (2.53). This holds also for countable state CTMCs,and if the CTMC is irreducible there is the additional guarantee that a solution of (2.53)-(2.54) (ifit exists) is unique.

Moreover the following properties relate the existence and form of the stationary distributionof the irreducible CTMC X with those of its embedded DTMC X and its embedded uniformizedDTMC X (if X is uniformizable):

• The CTMC possesses a stationary distribution p if and only if the corresponding embeddedDTMC possesses a stationary distribution π satisfying

∑

j∈S

πj

rj< ∞ (2.55)

in which case

pj =πj

rj

/∑

i∈S

πi

ri, j ∈ S. (2.56)

• If the CTMC is uniformizable, then it possesses a stationary distribution p if and only ifthe corresponding embedded uniformized DTMC possesses a stationary distribution π andmoreover p = π (irrespective of the uniformization rate).

A finite state CTMC X is necessarily uniformizable; and if, moreover, X is irreducible then X,X, and X possess unique stationary distributions and (2.55) is necessarily satisfied. Thus, for finitestate irreducible CTMCs (the most emphasized case in the course) we get the results presented inthe following theorem.

Theorem 3 If the finite state CTMC X is irreducible, then it possesses a unique stationary dis-tribution p, which is the unique solution to either one of the following sets of conditions:

pQ = 0 and p1 = 1 (2.57)

pP = p and p1 = 1 (2.58)

pj =πj

rj

/∑

i∈S

πi

ri, j ∈ S, where πP = π and π1 = 1. (2.59)

66

Example 32 Consider the CTMC X of Example 28, having state space {0, 1} and generatormatrix

Q =[−λ λ

µ −µ

].

The system pQ = 0 is equal to{−λp0 + µp1 = 0λp0 − µp1 = 0

so that its two equations are equivalent and, moreover, their common solution is p1 = λµp0. Using

this relation, we have

p0 + p1 = 1 =⇒(

1 +λ

µ

)p0 = 1 =⇒ p0 = 1/

(1 +

λ

µ

)=⇒ p0 =

µ

λ + µ.

Thus, in view of (2.57), the CTMC X possesses stationary distribution

[p0 p1

]=

[µ

λ+µλ

λ+µ

].

As an exercise, compute the stationary distribution of X using (2.58) and using (2.59). ¥

Example 33 Consider the finite state and irreducible CTMC X = {X(t), t ≥ 0} of Example 27,where X(t) denotes the number of produced components that have not been packed at time t.

As the embedded uniformized transition probability matrix with uniformization rate q = 4

P =

1/2 1/2 0 01/2 0 1/2 00 1/2 0 1/20 0 1/2 1/2

is a double stochastic matrix (i.e., it is a stochastic matrix whose column sums are equal to 1), thenthe stationary probability vector associated to P is the uniform vector π =

[1/4 1/4 1/4 1/4

].

In view of (2.58), the CTMC X possesses uniform stationary distribution on {0, 1, 2, 3}. As anexercise, compute the stationary distribution of X using (2.57) and using (2.59). ¥

Example 34 Consider the CTMC X of the (1, 3) inventory system described in Example 30, whichhas state space S = {0, 1, 2, 3}. Its vector of rates out of states r and its embedded transitionprobability matrix P are given by

r =

0.50.10.10.1

, P =

0 0 0 11 0 0 00 1 0 00 0 1 0

.

As P is a double stochastic matrix, then the stationary probability vector of the embedded DTMCis the uniform vector π =

[1/4 1/4 1/4 1/4

]. In view of (2.59), and since

3∑

i=0

πi

ri=

14

(1

0.5+

10.1

+1

0.1+

10.1

)=

324

= 8,

67

X possesses stationary distribution

p =[p0 p1 p2 p3

]=

14

[1

0.51

0.11

0.11

0.1

]/8 =

[116

516

516

516

].

As an exercise, compute the stationary distribution of X using (2.57) and using (2.58). ¥

The generic problem we address in the section is the following.

Problem 13 Given an irreducible CTMC, compute its stationary distribution if it exists and, ifso, derive quantities associated with the long-run (expected long-run) occupation of states, and long-run (expected long-run) cost/reward rates associated with sojourn times in states and transitionsbetween different states. In addition compute the limit distribution of X and associated measures.¥

The type of cost/reward functions {C(t), t ≥ 0}, where C(t) is the cost/reward incurred/obtainedon the interval of time [0, t] is of the form

C(t) =∫ t

0cX(s) ds +

N(t)∑

n=1

Yn(Xn−1, Xn) (2.60)

where N(t) is the number of transitions made by X on the interval of time [0, t] and the randomvariables Yn(j, k), for n ∈ IN and j, k ∈ S, are independent. Here cj denotes the cost/rewardrate per unit time that is incurred/obtained continuously when the CTMC is in state j. Similarly,Yn(j, k) represents the cost/reward incurred/obtained at the n-th transition of the uniformizedchain given that a transition from state j to state k takes place, and we let

cjk = E[Yn(j, k)]

for j 6= k, and assume that the coefficients cjk are finite constants. Moreover, we let cjj = 0 for allj ∈ S. The constant cjk denotes the expected value of the cost/reward incurred/obtained when atransition from state j to state k takes place, which is independent of the index of the transition.

We let c = [cj ]j∈S denote the column vector of cost rates incurred continuously in the states ofX and C = [cjk]j,k∈S be the matrix of expected costs associated with state transitions. To be ableto solve instances of the generic problem stated, we will resort to the following important result.

Theorem 4 Let X be an irreducible CTMC with state space S possessing (unique, and guaranteedto exist if X has finite state space) stationary distribution p = [pj ]j∈S. Then:

• The stationary distribution of X is its limit distribution; i.e.,

limt→+∞P(X(t) = j|X(0) = i) = pj (2.61)

for all i, j ∈ S.

• The long-run and the expected long-run proportion of time X spends in state j are equal topj; i.e.,

limt→+∞

1t

∫ t

01{X(s)=j} ds = lim

t→+∞E[1t

∫ t

01{X(s)=j} ds

]= pj (2.62)

for j ∈ S, regardless of the initial distribution.

68

• The long-run and expected long-run cost/reward rate per unit of time for cost/reward functionsof the form (2.60) is equal to pc + p[R • C]1, i.e.,

limt→+∞

C(t)t

= limt→+∞E

[C(t)

t

]=

∑

j∈S

pj cj +∑

j,k∈S

pj rjk cjk = pc + p[R • C]1 (2.63)

provided∑

j∈S pj |cj |+∑

j,k∈S pjrjk|cjk| < ∞ (this condition holds necessarily if X has finitestate space). ¥

Note that the products pjrjk appearing in (2.63) are the long-run transition rates from state jto state k. These long-run run transition rates are then multiplied by the expected costs/rewardsassociated with the transitions to obtain long-run (expected long-run) costs rewards. Similarly, thecost/reward rates cj are multiplied by the expected amount of time spent in state j per unit timepj to get the expected cost/reward accumulated continuously during one unit of time.

Example 35 We will compute the long-run revenue per hour for the CTMC X of the (1, 3) inven-tory system described in Example 30, with the cost structure given there. Thus, X has state space{0, 1, 2, 3} and rate matrix R, stationary vector p, vector of reward rates c and matrix of expectedrewards associated with transitions C given by

R =

0 0 0 0.50.1 0 0 00 0.1 0 00 0 0.1 0

, p =

[116

516

516

516

], c =

0−0.1−0.2−0.3

, C =

0 0 0 −540 0 0 00 40 0 00 0 40 0

.

Thus,

pc + p[R • C]1 =[

116

516

516

516

]

0−0.1−0.2−0.3

+

0 0 0 −2.54 0 0 00 4 0 00 0 4 0

1111

= 3.40625

and so, the long-run revenue per hour is equal to $ 3.40625. ¥

2.8 Time-reversibility

An irreducible CTMC X with state space S, rate matrix R, and stationary distribution p is time-reversible if

pi rij = pj rji (2.64)

for all i, j ∈ S with i 6= j (or simply for i < j, by symmetry). These equations are called thedetailed balance equations or reversibility equations. They assert that for any pair of states (i, j),the long-run rate of transitions from state i to state j equals the correspondent long-run rate oftransitions from state j to state i. They imply that under stationarity the CTMC looks the samewhen seen forward in time as when seen backward in time.

An important result is that if X is an irreducible CTMC with state space S and rate matrixR, then: if there is a probability row vector p such that the equations (2.64) are satisfied for alli, j ∈ S, then p is the stationary probability row vector of X and X is time-reversible. This last

69

result follows since if p is a probability row vector satisfying (2.64) then, by summing the equationswith i fixed and j taking all values different from i, we get

pi

∑

j 6=i

rij =∑

j 6=i

pj rji ⇔ pi ri =∑

j 6=i

pj rji

for i ∈ S, where ri is the rate out of state i. Thus, p solves the (global) balance equations (2.53)and so it is the stationary probability row vector of X. Thus, for an irreducible CTMC X:

• If X has (known) stationary distribution p, then we can check if X is time-reversible bychecking if the pair (p, R) satisfies the reversibility equations (2.64).

• If it is not known if X has stationary distribution, it may be checked if X is time-reversibleby checking if there is a probability row vector p such that the pair (p, R) satisfies thereversibility equations (2.64).

The last way to check reversibility is only advisable if most of the entries of the rate matrixare null. Note also that if the positive entries of the rate matrix are not distributed symmetrically(with respect to the main diagonal), then the CTMC may not be reversible. That is, if X istime-reversible, then rij > 0 if and only if rji > 0. Thus, e.g., the CTMC of Example 35 is nottime-reversible.

The main class of time-reversible CTMCs we will be interested in this course are (stationary)birth-death processes, mainly because:

An irreducible birth-death process is stationary if and only if it is time-reversible.

To illustrate how this result may be concluded, consider that X is an irreducible birth-deathprocess with state space

S = {0, 1, 2, . . . , M − 1,M}(here it is possible that M = +∞) with birth-rates {λi, i = 0, 1, . . . ,M − 1} and death-rates{µi, i = 1, 2, . . . , R}. The balance equations (2.53) are

{λ0p0 = µ1p1

λi−1pi−1 + µi+1pi+1 = (λi + µi)pi, 1 ≤ i < M(2.65)

where we have dropped the equation for state M (if M is finite) – since each one of the globalbalance equations is consequence of the remaining ones. Moreover, the reversibility equations are

λipi = µi+1pi+1, 0 ≤ i < M. (2.66)

Thus, the reversibility equation is satisfied for i = 0. Moreover, if we assume that it is satisfied fori < k with 0 < k < M , i.e., λipi = µi+1pi+1 for i = 0, 1, . . . , k − 1. Then, as in view of (2.65),

λk−1pk−1 + µk+1pk+1 = (λk + µk)pk.

and by induction hypothesis λk−1pk−1 = µkpk, it follows that

µk+1pk+1 = λkpk

which corresponds to (2.66) for i = k. Thus, it follows that (2.66) holds for 0 ≤ i < M , i.e., X istime reversible.

70

The main consequence of the previous result is that the reversibility equations (2.66) may beused to compute the stationary distribution of an irreducible birth-death process whenever it exists.As the equations (2.66) may be rewritten in the form

pi+1 =λi

µi+1pi, i = 0, 1, . . . , M − 1 (2.67)

they lead, using induction, to

pi =

[i−1∏

k=0

λk

µk+1

]p0, i = 1, 2, . . . ,M. (2.68)

Now, the birth-death process will have a stationary distribution p =[p0 p1 . . . pM

]if

M∑

i=0

pi = 1 ⇐⇒M∑

i=0

[i−1∏

k=0

λk

µk+1

]p0 = 1 ⇐⇒ p0 =

[M∑

i=0

[i−1∏

k=0

λk

µk+1

]]−1

> 0

(since if p0 = 0 then pi = 0 for i = 0, 1, . . . , M , and p would not be a probability vector), so thatwe get the following important result.

Theorem 5 The irreducible birth-death process X with state space S = {0, 1, 2, . . . , M}, birth ratesλi, i = 0, 1, . . . , M −1, and death rates µi, i = 1, 2, . . . ,M , has a stationary distribution if and onlyif

M∑

i=0

[i−1∏

k=0

λk

µk+1

]< ∞. (2.69)

Moreover, if it exists, the stationary distribution of X, p =[p0 p1 . . . pM

]is given by

pi =

∏i−1k=0

λkµk+1∑M

j=0

[∏j−1k=0

λkµk+1

] (2.70)

for i = 0, 1, 2, . . . , M .

Note that, in particular, all irreducible birth-deaths processes with finite state space have a(unique) stationary distribution, given by (2.70), and are time-reversible. Although we will be ableto decide if an irreducible CTMC is time-reversible, in general, we will be mainly concerned in thissection with the following problem.

Problem 14 Given an irreducible birth-death process, decide if it has a stationary distributionand, if yes, compute its stationary distribution. ¥

To address the stated problem we only need to use Theorem 5. In this section we only provideone example of its use. More examples will be seen in the next chapter, along with the homeworksand tests.

Example 36 Consider the CTMC X = {X(t), t ≥ 0} of Example 27 and Example 33. X(t)denotes the number of produced components that have not been packed at time t, for t ≥ 0, andX is a time-reversible birth-death process with rates λ0 = λ1 = λ2 = µ1 = µ2 = µ3 = 2 and so

λ0

µ1=

λ1

µ2=

λ2

µ3=

22

= 1.

71

Thus,∑3

i=0

[∏i−1k=0

λkµk+1

]= 1 + 1 + 1 + 1 = 4 and, using (2.70), X has stationary distribution

p =[p0 p1 p2 p3

]=

[14

14

14

14

]. ¥

2.9 Closing notes

We note that (2.55)-(2.56) was the least emphasized way to compute the stationary distribution of aCTMC, because it is the least efficient of the three presented methods. However, this method is veryeasily generalizable to perform computations for semi-Markov processes. Semi-Markov processeshave similar behavior to CTMCs with the exception that the sojourn times in states do not haveexponential distribution.

Thus, as for CTMCs, in a semi-Markov process X = {X(t), t ≥ 0} with countable state spaceS, the sequence of successive states visited by X, X = {Xn, n ≥ 0} is a DTMC and the sojourntimes in states do not depend on the previous states visited and the times spent on those states.Moreover all sojourn times in a state are identically distributed. Then, if each sojourn time instate i ∈ S has mean 1/ri, and X is an irreducible DTMC with finite state space and transitionprobability matrix P , then (2.62) and (2.63) hold with rjk substituted by rj pjk.

Example 37 Consider Example 22 and assume that, instead of the cash register machine beingoperational and under repair for exponentially distributed amounts of time, we just know that theexpected value of the repair time (in days) is 1/r0 = 1/0.90 = 10/9 and the expected time themachine is operational is 1/r1 = 1/0.01 = 100. If, in addition, the successive repair times andperiods in which the machine is operational are independent and we let

X(t) =

{1 the cash register machine is operational at time t

0 the cash register machine is under repair at time t

then X = {X(t), t ≥ 0} is a semi-Markov process, having embedded transition probability matrix(at transition epochs)

P =[0 11 0

].

As the stationary probability vector associated to P is π =[1/2 1/2

], the long-run fraction of

time the cash-register machine is operational is

p1 =π1r1

π0r0

+ π1r1

=1/2

1/0.01

1/21/0.90 + 1/2

1/0.01

=0.90

0.01 + 0.90=

9091

and the long-run rate (per day) at which the machine goes under repair is

p1r1p10 = p1r1 =9091× 0.01 =

9910

. ¥

72

Chapter 3

Queueing Networks

In this chapter we will study queues and queueing networks on continuous time (there are analogousdiscrete time models). Our analysis will rely mainly in the general theory of CTMCs. However wewill introduce several concepts and properties specific of queueing theory.

Queues arise naturally in systems where service is requested by customers (these may be, e.g,humans at a counter, partially processed parts at a manufacturing facility, packets to be transmittedat a telecommunications switch). The service of customers is performed at one or more queue stationand each queue station may have either a single server or multiple servers. When arriving to thesystem (queueing system or queueing network) a customer may not be able to start being servedimmediately, in which case it may be allowed to wait in a queue. In manufacturing systems, asynonym for queueing system is inventory system. Moreover, when there are more than one queuestation (i.e., in queueing networks) after being serviced at one station a customers may requestservice at other station.

There is a great number of types of queueing networks. In broad terms, the description of aqueuing network involves the specification of:

• The network topology, i.e., the number and physical characteristics of the queue stations(including characteristics like maximum queue size, server(s) failure(s), speed of the server(s)at the station, etc.);

• The external arrival process of customers (for open queueing networks – i.e., networks inwhich customers enter the network from the exterior);

• The routing policy, describing the way customers are routed among the server stations afterarriving to the network; and

• The service policy, describing the way servers choose which customers to serve or the waythey distribute service among the customers present at each station.

3.1 Single-station queueing systems

First we will look at (open) single-station queueing systems where customers arrive (one at a time)from the exterior, are serviced at the station (if allowed to join the system), and after being serviceddepart from the system. Moreover, we assume that

73

• The interarrival times of customers are independent and identically distributed positive ran-dom variables (i.e., the arrival process of customers is a renewal process) with finite expectedvalue. The distribution of the interarrival times is denoted by the following letters: M forexponential, D for deterministic, Ek for Erlang with k phases, and G for general distribution.

• The service times of customers are independent and identically distributed positive randomvariables. Their common distribution is denoted by the same letters as the distribution ofthe interarrival times. Moreover, the sequences of interarrival times and service times areindependent.

• There are s identical servers that can serve any customer and serve only one customer at atime. A server can be idle only if there are no customers waiting for service.

• The system capacity is s+ c, i.e., in addition to a maximum of s customers being served theremay be at most c customers waiting for their service to be initiated (the queue capacity is c).Thus, if when arriving to the system a customer finds s+ c customers in the system he is notallowed to join the system and is lost (or blocked).

We will use the notation A/B/c/d to denote a (single-station) queueing system as describedabove having: interarrival time distribution A, service time distribution B, s servers, and capacityd. When d is infinite we may drop it from the notation and denote the system by A/B/c. Moreoverwe will assume that the service of customers is initiated by their order of arrival to the system; i.e.,the First Come First Served (FCFS) service policy is used.

We next introduce some notation for important primary parameters of single-station queueingsystems:

• Arrival Rate (λ): the arrival rate is the long-run mean number of customers that arrive tothe system per time unit. If we let Y1, Y2, . . . denote the sequence of interarrival times andN(t) denote the number of customer arrivals until time t, then

λ = limt→∞

N(t)t

=1

E[Y1](3.1)

so that the interarrival times have expected value 1/λ (also, in the long-run, we should expecton average one arrival per each 1/λ units of time, provided the interarrival time distributionis aperiodic).

• Service Rate (µ): the service rate (per single server) is the reciprocal of the expected servicetime of a customer. If we let S1, S2, . . . denote the service times of customers 1, 2, . . ., then

µ =1

E[S1](3.2)

and µ equals the long-run mean number of customers that a server would serve per time unitif the server was always busy.

• (Offered) Load (a = λ/µ): the load denotes the expected amount of time a (single) serverwould take to serve all customers that in the long-run arrive to the system during one unit oftime, including blocked customers. The offered load has no units but it is sometimes expressedin Erlangs.

74

• (Offered) Traffic Intensity (ρ = λ/(sµ)): the traffic intensity is a (relative) measure of con-gestion and represents the load offered to each server if the work is divided equally amongservers.

Note that if the arrival process to the single-station queueing system is a Poisson process withrate λ, then the arrival rate to the system is λ, as it should be!

Some derived or performance parameters of single-station queueing systems are:

• Blocking Probability (Pb): the blocking probability is the long-run fraction of customers thatare blocked.

• Input Rate (λe = λ[1 − Pb]): the input rate is the long-run mean number of customers thatenter the system per time unit, thus excluding blocked customers.

• Carried Load (ae = λeµ = λ

µ [1 − Pb] = a[1 − Pb]): the carried (or effective) load denotesthe expected amount of time a (single) server would take to serve all customers that in thelong-run enter the system during one unit of time, thus excluding blocked customers.

• Carried Traffic Intensity (ρe = λsµ [1 − Pb] = ρ[1 − Pb]): the carried (or effective) traffic

intensity represents the effective load carried by each server if the work is divided equallyamong servers.

• Throughput rate: the throughput rate is the rate at which customers complete their serviceat the system, i.e., the long-run mean number of served customers that leave the system pertime unit. With our conditions, the throughput rate is

{λe c < ∞min(λ, sµ) c = ∞ (3.3)

as we will be seen later. Thus, if the capacity is finite or the traffic intensity is strictly lessthan one (λ < µ), then the throughput rate equals the input rate (that is, rate in = rate out).

• Long-run fraction of time in state i (pi): for i = 0, 1, 2, . . ., pi denotes de long-run fraction oftime there are i customers in the system.

Note that if the capacity is infinite, c = +∞, then the input rate (carried load, carried trafficintensity) equals the arrival rate (offered load, offered traffic intensity) as no customers are blocked,i.e., Pb = 0.

Throughout the chapter we let Q(t) denote the number of customers in the (queueing) systemat time t. We will say that the queueing system is stable if the set of long-run fraction of time instates constitutes a probability distribution, i.e.,

s+c∑

i=0

pi = 1. (3.4)

Note that if Q = {Q(t), t ≥ 0} is an irreducible CTMC then {pi, i = 0, 1, . . . , s+c} is the stationaryor steady-state distribution of Q.

Aside from the fraction of time the system spends in each state, other occupancy related dis-tributions of interest include the fraction of customers that: find i customers when arriving to thesystem (includes customers that are blocked), find i customers when entering the system (does not

75

include blocked customers), leave behind i customers in the system at departure time. The lasttwo coincide for all stable systems with single arrivals, but they are different in general from thedistribution of the number of customers seen by an arriving customer for finite capacity systems.

We will study in detail M/M/s/s + c systems, i.e., systems with Poisson (process) arrivalsand independent and identically distributed exponential service times. For these systems all thementioned distributions can be easily derived from the long-run fraction of time the system spendsin each state. Moreover, these systems possess the PASTA (Poisson Arrivals See Time Averages)property: the long-run fraction customers that find i customers in the system coincides with thefraction of time the system spends in state i, for i = 0, 1, . . . , s + c.

Aside from occupancy related distributions, other distributions of interested are the long-rundistribution of the sojourn (waiting) time of a customer in the system (queue). These distributionsare more related with the point of view of customers, viz. customer satisfaction, whereas occupancyrelated distributions are more related with the point of view of resource management.

3.2 Stability of M/M/s/s + c systems

In this section we study M/M/s/s + c systems and use the notation introduced in the previoussection. That is, we consider a (single-station) queueing system with s identical servers, queuecapacity c, Poisson customer arrival process with rate λ, and exponential service times with rate µ.

Using the additional assumptions on the independence of service times and the independenceof the sequences of interarrival and service times, it follows that the process Q, where as mentionedbefore Q(t) denotes de number of customers in the system at time t, is a CTMC. More precisely,Q is a birth-death process with state space S = {0, 1, 2, . . . , s + c} and arrival and service rates:

λi = λ, i = 0, 1, . . . , s + c− 1 and µi =

{iµ i = 1, 2, . . . , s− 1sµ i = s, s + 1, . . . , s + c

. (3.5)

This follows since:

· The customer interarrival times are independent random variables with exponential distribu-tion with rate λ and customers that arrive to the system when there are s + c customers inthe system do not enter the system;

· When the number of customers in the system, i, is smaller than the number of servers, thenthe number of busy servers is equal to the number of customers in the system – each busyserver is serving one of the customers in the system – and, since the different service timesare independent and have exponential distribution with parameter µ, the amount of time forthe first service to be finished has exponential distribution with parameter iµ; and

· When there are s or more customers in the system all s servers are busy and thus, using againthe same argument, the amount of time for the first service to be finished has exponentialdistribution with parameter sµ.

As a consequence, all the results presented in Chapter 2 may be applied to M/M/s/s + c systemsincluding, in particular, transient and long-run cost/reward analysis associated with the processQ of the number of customers in the system. In addition, note that (if the number of servers isfinite, as assumed unless otherwise explicitly stated) Q is uniformizable as the rates out of statesare bounded by (λ + sµ).

76

We next derive the conditions for stability of M/M/s/s+ c systems. From (3.5), it follows that

λk

µk+1=

{λ/[(k + 1)µ] k = 0, 1, . . . , s− 1λ/(sµ) k = s, s + 1, . . . , s + c− 1

=

{(sρ)/(k + 1) k = 0, 1, . . . , s− 1ρ k = s, s + 1, . . . , s + c− 1

and, thereforei−1∏

k=0

λk

µk+1=

{(sρ)i/i! i = 0, 1, . . . , s− 1(sρ)sρi−s/s! i = s, s + 1, . . . , s + c− 1

.

As a result,s+c∑

i=0

i−1∏

k=0

λk

µk+1=

s−1∑

i=0

(sρ)i

i!+

(sρ)s

s!

c∑

k=0

ρk.

As,∑c

k=0 ρk is finite if c is finite or ρ < 1, and is infinite otherwise (i.e., c = ∞ and ρ ≥ 1) weconclude, in view of Theorem 5, the following result.

Theorem 6 The system M/M/s/s+ c is stable if and only if c is finite or ρ < 1. Moreover, if thesystem is stable then it possesses steady-state distribution

pi =

(sρ)i

i!Ps−1j=0

(sρ)j

j!+

(sρ)s

s!

Pck=0 ρk

i = 0, 1, . . . , s− 1

(sρ)s

s!ρi−s

Ps−1j=0

(sρ)j

j!+

(sρ)s

s!

Pck=0 ρk

i = s, s + 1, . . . , s + c. (3.6)

In addition, the system in steady-state is time reversible.

An important fact is that the steady-state distribution of the number of customers in (stable)M/M/s/s + c systems is only function of the number of servers s, the queue capacity c, and thetraffic intensity ρ. Thus, the specific values of the arrival rate λ and the service rate µ are notimportant for effect of the steady-state distribution of the number of customers in the system (or inthe queue); however, they are important for transient distributions and for the steady-state sojourn(waiting) time of customers in the system (queue). From now onwards we will consider only stableM/M/s/s + c systems even if not mentioned explicitly.

One consequence of the previous theorem is that since an M/M/s system in steady state hasPoisson arrival process and is time reversible, and since an arrival in the reversed-time processcorrespond to a departure in the original process, the departure process from an M/M/s systemin steady state is a Poisson process (with the same rate as the arrival process).

For an M/M/s/s+ c system we let Ls (Lq) denote the steady-state number of customers in thesystem (queue), and Ws (Wq) denote the steady-state sojourn (waiting) time of a customer in thesystem (queue). Note that, in particular, Ls has probability function p =

[p0 p1 . . . ps+c

], i.e.,

pi = P(Ls = i) and, for computations involving these probabilities,

n∑

k=0

ρk =

11−ρ n = ∞, ρ < 11−ρn+1

1−ρ n < ∞, ρ 6= 1

n + 1 n < ∞, ρ = 1

. (3.7)

77

3.3 Blocking probabilities and loss systems

The blocking probability Pb is the long-run fraction of customers that are blocked. In M/M/s/s+csystems with infinite capacity blocking does not occur, but in systems with finite queue capacityc blocking occurs whenever a customer finds s + c customers in the system when arriving to thesystem. Thus, the blocking probability – denoted by B(s, sρ, c) – for finite capacity M/M/s/s + csystems is just ps+c = P(Ls = s + c) and is given by

B(s, sρ, c) =(sρ)s

s! ρc

∑s−1j=0

(sρ)j

j! + (sρ)s

s!

∑ck=0 ρk

. (3.8)

If ρ < 1, we understand the block probability B(s, sρ, +∞) as being zero. Some properties of theblocking probability B(s, sρ, c) are:

· It decreases with the queue capacity c;

· It decreases with the number of servers; and

· It increases with the offered traffic intensity (or the offered load).

In these statements the parameters not mentioned explicitly in each sentence are assumed as beingfixed.

The blocking probability formula (3.8) may be used to choose appropriately the parametersof the system so that some low blocking probability is achieved; this involves usually determining(a sufficiently large) queue capacity c but may involve also, e.g., the computation of an adequatenumber of servers s.

Example 38 For the system M/M/1/1 + c, the blocking probability is

B(1, ρ, c) =ρc+1

1 + ρ∑c

k=0 ρk=

ρc+1

∑c+1k=0 ρk

=

{ρc+1(1−ρ)1−ρc+2 ρ 6= 11

c+2 ρ = 1. (3.9)

Note that if ρ = 1, then the blocking probability is (c + 2)−1. Moreover, if ρ > 1, then using (3.9)we may conclude that

B(1, ρ, c) ≥ λ− µ

λ= 1− 1

ρ(3.10)

i.e., 1− 1/ρ is a lower bound for the blocking probability. We may look at µ as an upper bound forthe rate at which customers enter the system, since the rate at which customers enter the systemis equal to the rate at which customers leave the system and the rate at which customers leavethe system can not be larger than µ as the service rate is µ. This, in turn, implies that λ − µ isa lower bound for the rate at which customers are blocked, so that (λ− µ)/λ is a lower bound forthe long-run fraction of blocked customers, i.e., the blocking probability.

Thus, for ρ > 1, the blocking probability can not be made arbitrarily small by increasing thequeue capacity. In that situation either more servers or a faster server may be needed to achievesome desired low blocking probability. Conversely, if ρ ≤ 1, then any arbitrarily low blockingprobability may be achieved by choosing an appropriate queue capacity c. For example if ρ = 0.9,then the minimum queue capacity c for which the blocking probability B(1, 0.9, c) is smaller than1% is 22 since B(1, 0.9, 21) = 1.08% but B(1, 0.9, 22) = 0.96 %. ¥

78

The generalization of the property (3.10) to an arbitrary number of servers leads to

B(s, sρ, c) ≥ λ− sµ

λ= 1− 1

ρ(3.11)

for ρ > 1. Moreover the blocking probability may be made as close as we want to the lower bound1− 1/ρ by increasing enough the queue capacity c.

The throughput rate of the M/M/s/s + c system (here c may be infinite provided the systemis stable) is equal to the input rate,

λe = λ[1−B(s, sρ, c)]. (3.12)

Thus, if c = ∞ then λe = λ, but if c < ∞ then λe < λ since a positive fraction of customers areblocked. Moreover the steady-state number of busy servers (which is equal to the difference of thenumber of customers in the system and the number of customers in the queue) has expected value

E[Ls − Lq] = sρ[1−B(s, sρ, c)] (3.13)

and, thus, is equal to the carried load (ae) or s times the carried traffic intensity (sρe).The systems M/M/s/s are particular cases of loss systems, i.e., systems where no waiting in

queue is allowed. In this systems a customer joins the system only if there is at least one ide serverwhen he arrives to the system and is lost (or blocked) otherwise. The steady-state probabilities ofthe number of customers in the system are

pi =(sρ)i

i!∑sj=0

(sρ)j

j!

(3.14)

for i = 0, 1, . . . , s. Thus, it follows that Lsd= (X|X ≤ s) with X ∼ P (sρ), i.e., the steady-state

distribution of the number of customers in the M/M/s/s loss system is equal to the conditionaldistribution of a Poisson random variable with parameter sρ given that it is smaller or equal to s,in short, a Poisson random variable truncated at s.

Note that for M/M/s/s loss systems the steady-state distributions of the number of customersin the queue and the sojourn (waiting) time of a customer in the system (queue) are the easiestto obtain among all M/M/s/s + c systems. Specifically, P(Lq = 0) = P(Wq = 0) = 1 andWs ∼ Exp(µ); i.e., there is no queueing and so the sojourn time of a customer in the system isequal to its service time, which has exponential distribution with parameter µ. For these systems,the blocking probability

B(s, sρ, 0) =(sρ)s

s!∑sj=0

(sρ)j

j!

(3.15)

(usually denoted simply by B(s, sρ)) which is called Erlang blocking formula, Erlang blocking prob-ability, or Erlang’s first formula, plays a major role.

The Erlang blocking probability was the first queueing formula that played a major role inapplications. In the planning of loss systems a major objective is to guarantee maximum serverutilization (i.e., minimum number of servers) constrained to a given upper bound on the blockingprobability (which may be viewed as a minimum Quality of Service (QoS) level). This constitutedan important engineering problem in the planning of the early telephone networks where: dedicatedlines were assigned to calls when unused lines were available, and otherwise calls were blocked.

79

3.4 Loss-delay probabilities

The loss-delay probability is the long-run fraction of customers that are either blocked or delayed.Thus, the loss-delay probability in M/M/s/s+c systems, denoted by C(s, sρ, c), is just the steady-state probability that a customer finds all s servers busy when arriving to the system, P(Ls ≥ s) =∑s+c

k=s pk, given by

C(s, sρ, c) =(sρ)s

s!

∑ck=0 ρk

∑s−1j=0

(sρ)j

j! + (sρ)s

s!

∑ck=0 ρk

. (3.16)

Similarly to the blocking probability B(s, sρ, c), the loss-delay probability C(s, sρ, c) enjoys thefollowing nice properties:

· It increases with the maximum queue capacity c;

· It decreases with the number of servers; and

· It increases with the offered traffic intensity (or the offered load).

Example 39 For the system M/M/1/1 + c, with c < ∞, the loss-delay probability is

C(1, ρ, c) =ρ

∑ck=0 ρk

1 + ρ∑c

k=0 ρk=

∑c+1k=1 ρk

∑c+1k=0 ρk

=

{ρ−ρc+2

1−ρc+2 ρ 6= 1c+1c+2 ρ = 1

. (3.17)

Note that if ρ = 1 then the loss-delay probability is (c + 1)/(c + 2), which is close to 1 if the queuecapacity is large. ¥

For systems with infinite queue capacity, also called delay systems, i.e., systems M/M/s, theloss-delay probability becomes

C(s, sρ) = C(s, sρ, +∞) =(sρ)s

s!(1−ρ)∑s−1j=0

(sρ)j

j! + (sρ)s

s!(1−ρ)

(3.18)

and is called Erlang delay formula or Erlang’s second formula. We will call C(s, sρ) the delayprobability since in delay systems no customers are lost.

Example 40 The delay probability in the M/M/1 system is

C(1, ρ) =ρ/(1− ρ)

1 + ρ/(1− ρ=

ρ

1− ρ + ρ= ρ.

Thus, the delay probability in the M/M/1 system equals: the traffic intensity, the long-run fractionof time the server is busy, and the steady-state expected number of busy servers.

Similarly, the delay probability in the M/M/2 system is

C(2, 2ρ) =2ρ2/(1− ρ)

1 + 2ρ + 2ρ2/(1− ρ)=

2ρ2

(1 + 2ρ)(1− ρ) + 2ρ2=

2ρ2

1 + ρ.

Note that

C(2, 2ρ)− C(1, ρ) =2ρ2

1 + ρ− ρ =

ρ2 − ρ

1 + ρ=

ρ(ρ− 1)1 + ρ

< 0.

This does not constitute a surprise since, as mentioned before, the delay probability decreases withthe number of servers, for fixed traffic intensity. ¥

80

3.5 Steady-state number of customers in queue

In this section we address the computation of the steady-state number of customers in queue forM/M/s/s + c systems, Lq. We first note that

Lq =

{0 Ls < s

Ls − s Ls ≥ s. (3.19)

Therefore, the steady-state number of customers in the queue has probability function

P(Lq = k) =

{∑sj=0 pi k = 0

ps+k k = 1, 2, . . . , c(3.20)

where the pj = P(Ls = j) are given by (3.6). The probability function given above can thus beused to derive all characteristics of the steady-state number of customers in queue for M/M/s/s+csystems.

However, we will next provide an alternative characterization of the steady state number of cus-tomers in queue in terms of mixtures of random variables and the modified geometric distribution,both of which we will next introduce.

3.5.1 The modified geometric distribution

We say that a random variable Y has a modified geometric distribution with parameters r, r ∈IN ∪ {+∞}, and p, p < 1, and write Y ∼ modG(r, p), if

P(Y = k) =(1− p)k

∑rj=0(1− p)j

(3.21)

for k = 0, 1, . . . , r, where p ≥ 0 if r = +∞.If r = +∞ we may write simply Y ∼ modG(p) and Y may be interpreted as the number of

failures occurring in a sequence of independent and identically distributed Bernoulli trials withsuccess probability p strictly before the occurrence of the first success, and then

P(Y = k) = (1− p)kp (3.22)

for k ∈ IN0. In this case

E[Y ] =1− p

p, Var[Y ] =

1− p

p2. (3.23)

In the case where r is finite, it may be conclude that

E[Y ] =

{1−p

p × 1−[1+rp](1−p)r

1−(1−p)r+1 p 6= 0r2 p = 0

. (3.24)

The formula for the variance of Y may also be derived but it is more cumbersome.

81

3.5.2 Mixtures of random variables

We say that the random variable W is a random mixture of the random variables Y and Z,

W = pY ⊕ (1− p)Z

where p is a probability, if with probability p the random variable W is made equal to Y andotherwise W is made equal to Z. Thus, in particular, P(W = Y ) = p = 1−P(W = Z).

An important result for mixtures, that can be easily derived and which we will use, is that ifW = pY ⊕ (1− p)Z, then

E[W ] = pE[Y ] + (1− p)E[Z] (3.25)

Var[W ] = pVar[Y ] + (1− p)Var[Z] + p(1− p)(E[Y ]−E[Z])2. (3.26)

The simplest case of a mixture is the Bernoulli random variable which is a mixture of therandom variables that are equal to 1 and 0 with probability one. Since constant random variableshave null variance, it follows that the variance of a Bernoulli random variable with parameter p,p(1− p), may be obtained from (3.26).

3.5.3 Steady-state distribution of the number of customers in queue as a mix-ture

We next derive for an M/M/s/s+c system the steady-state distribution of the number of customersin queue given that all servers are busy, Lq|Ls ≥ s. This equals the distribution of (Ls − s|Ls ≥ s)in view of (3.19). As, using (3.6), we have, for i = 0, 1, . . . , c,

P(Lq = i|Ls ≥ s) =P(Ls = s + i)

P(Ls ≥ s)=

P(Ls = s + i)∑s+ck=s P(Ls = k)

=(sρ)s

s! ρi

(sρ)s

s!

∑ck=0 ρk

=ρi

∑ck=0 ρk

we conclude that

(Lq|Ls ≥ s) ∼ modG(c, 1− ρ). (3.27)

That is, the steady-state distribution of the number of customers in queue given that all serversare busy is a modified geometric random variable with parameters c and 1− ρ.

Moreover, as if not all servers are busy the queue is empty, then the steady-state distributionof the number of customers in queue given that not all servers are busy is the random variable thatis equal to zero with probability one.

As the steady-state probability that all servers are busy is the loss-delay probability C(s, sρ, c),it then follows that the steady-state distribution of the number of customers in queue is the mixture

Lqd= [1− C(s, sρ, c)]O ⊕ C(s, sρ, c)N?

c . (3.28)

where P(O = 0) = 1 and N?c ∼ modG(c, 1 − ρ). Thus, (3.28) says that with the loss-delay

probability, C(s, sρ, c), the steady-state number of customers in queue takes a value from a modi-fied geometric random variable with parameters c and (1 − ρ), N?

c , and with the complementaryprobability it takes the value zero.

82

Using (3.23), (3.25)-(3.26), (3.28), and the fact that E[O] = Var[O] = 0, we get that for theM/M/s system,

E[Lq] =ρ

1− ρC(s, sρ) (3.29)

Var[Lq] =ρ

(1− ρ)2C(s, sρ) (1 + ρ[1− C(s, sρ)]) . (3.30)

Note that, in particular, both the expected value and the variance of the steady-state number ofcustomers in the queue tend to infinity as ρ tends to 1. Similarly, using (3.24), we get that for theM/M/s/s + c system with finite queue capacity,

E[Lq] =[

ρ

1− ρ

1− (c + 1)ρc + cρc+1

1− ρc+1

]C(s, sρ, c) (3.31)

if ρ 6= 1, and E[Lq] = c2C(s, s, c) if ρ = 1.

Example 41 For the M/M/1 system, as the delay probability is equal to the traffic intensity, i.e.,C(1, ρ) = ρ, (3.29)-(3.30) lead to

E[Lq] =ρ2

1− ρ, Var[Lq] =

ρ2

(1− ρ)2[1 + ρ(1− ρ)] =

ρ2

(1− ρ)2+

ρ3

1− ρ. (3.32)

Similarly, from Example 40, we have that the delay probability in the M/M/2 system isC(2, ρ) = 2ρ2/(1 + ρ). Thus, using (3.29)-(3.30),

E[Lq] =2ρ3

1− ρ2, Var[Lq] =

2ρ3

(1− ρ)2(1 + ρ)

[(1 + ρ)− 2ρ3

1 + ρ

]=

2ρ3

(1− ρ)2−

(2ρ3

1− ρ2

)2

. (3.33)

Note that these formulas are similar to those obtained for M/M/1 systems but higher order poly-nomials on ρ arise. ¥

Example 42 Since for the system M/M/1/1+c the loss-delay probability is given by (3.17), using(3.31), we have

E[Lq] =[

ρ

1− ρ× 1− (c + 1)ρc + cρc+1

1− ρc+1

]ρ− ρc+2

1− ρc+2=

ρ2

1− ρ× 1− (c + 1)ρc + cρc+1

1− ρc+2(3.34)

if ρ 6= 1, and E[Lq] = c2

c+1c+2 = c(c+1)

2(c+2) if ρ = 1. ¥

3.6 Steady-state waiting time in queue

In this section we derive the steady-state distribution of the waiting time in queue, Wq. As onlycustomers that enter the system wait in queue, we need first to derive the steady-state distributionof the number of customers seen by entering customers.

83

3.6.1 Steady-state distribution of the number of customers seen by enteringcustomers

Due to the PASTA property, the steady-state distribution of the number of customers in the systemseen by arriving customers to a stable M/M/s/s + c system is just the steady-state distributionof the number of customers in the system. As an arriving customer enters the system if and onlyif there is one or more positions available in the system at its arrival instant, then the steady-state distribution of the number of customers seen by an entering customer is just the conditionaldistribution of Ls given that Ls < s + c, i.e., the distribution of (Ls|Ls ≤ s + c− 1).

As, for i = 0, 1, . . . , s + c− 1,

P(Ls = i|Ls ≤ s + c− 1) =P(Ls = i)

P(Ls ≤ s + c− 1)=

P(Ls = i)∑s+c−1k=0 P(Ls = k)

we conclude, using (3.6), that

The steady-state distribution of the number of customers seen by entering customers in anM/M/s/s + c system is the steady-state distribution of the number of customers in a

M/M/s/s + c− 1 system with the same arrival and service rate.

The previous statement is valid regardless of the M/M/s/s + c system having finite or infinitecapacity. If c = 0, the system M/M/s/s + c − 1, or M/M/s/s − 1, appearing in the statementshould be understood as a system with s servers such that at most s− 1 customers are allowed tobe simultaneously in the system.

Example 43 In an M/M/1/2 system with arrival rate 2 and service rate 3, the steady-statedistribution of the number of customers in the system, Ls, is

pi(M/M/1/2) =

9/19 i = 06/19 i = 14/19 i = 2

.

Similarly, the steady-state distribution of the number of customers in the M/M/1/1 system witharrival rate 2 and service rate 3 is

pi(M/M/1/1) =

{3/5 i = 02/5 i = 1

.

This last probability function is just the steady-state distribution of the number of customers inthe previous M/M/1/2 system conditional to being smaller than 2 (the capacity of the system). ¥

3.6.2 Steady-state distribution of the number of customers in queue as a mix-ture

The result presented in the previous subsection implies, in particular, that the steady-state prob-ability that an entering customer to an M/M/s/s + c system, with c ≥ 1, has to wait in queueequals the loss-delay probability for a system with queue capacity c− 1. This also follows from therelation

P(Ls ≥ s|Ls ≤ s+ c−1) =∑s+c−1

k=s P(Ls = k)∑s+c−1k=0 P(Ls = k)

=C(s, sρ, c)−B(s, sρ, c)

1−B(s, sρ, c)= C(s, sρ, c−1). (3.35)

84

In addition, from (3.27), we conclude that given that an entering customer to an M/M/s/s + csystem, with c ≥ 1, has to wait in queue, then the distribution of the number of customers he findsin queue has distribution modG(c − 1, 1 − ρ). Thus, the steady-state distribution of the waitingtime in queue, Wq, is given by the following mixture

Wqd= [1− C(s, sρ, c− 1)]O ⊕ C(s, sρ, c− 1)

N?c−1+1∑

i=1

Ei (3.36)

where the Eiiid∼ Exp(sµ) and represent times between terminations of services when all servers are

busy and, as before N?c−1 ∼ modG(c − 1, 1 − ρ). Moreover N?

c−1 and the random variables Ei areindependent. Note that if an entering customer finds all servers busy and k customers in queue hehas to wait for the termination of k + 1 services to start being served; this is the reason why thenumber of service completions that appear in (3.36) is N?

c−1 + 1.We now derive the distribution of the steady-state distribution of the waiting time in queue,

Wq. From (3.36), we haveP(Wq = 0) = 1− C(s, sρ, c− 1)

so that there is a positive probability that an entering customer does not have to wait in queue.Moreover by conditioning on the steady-state number of customers in the system seen by an enteringcustomer, we have, for t ≥ 0,

P(Wq > t) = C(s, sρ, c− 1)P

N?c−1+1∑

i=1

Ei > t

by (3.36)

= C(s, sρ, c− 1)c−1∑

k=0

P(N?c−1 = k)P

(k+1∑

i=1

Ei > t|N?c−1 = k

)by total probability law

= C(s, sρ, c− 1)c−1∑

k=0

ρk

∑c−1j=0 ρj

P(Erlang(k + 1, sµ) > t)

= C(s, sρ, c− 1)c−1∑

k=0

ρk

∑c−1j=0 ρj

k∑

n=0

e−sµt (sµt)n

n!by (2.7)

= C(s, sρ, c− 1)c−1∑

n=0

e−sµt (sµt)n

n!

∑c−1k=n ρk

∑c−1j=0 ρj

by rearranging terms.

Thus,

P(Wq > t) = C(s, sρ, c− 1)c−1∑

n=0

e−sµt (sµt)n

n!

∑c−1k=n ρk

∑c−1j=0 ρj

(3.37)

for t ≥ 0 and P(Wq > t) = 1 for t < 0 since Wq is a nonnegative random variable. The previousformula may be used to derive any characteristics of the the steady-state distribution of the waitingtime in queue, Wq.

85

For systems with infinite capacity, (3.37) may be simplified as follows

P(Wq > t) = C(s, sρ)∞∑

n=0

e−sµt (sµt)n

n!ρk by (3.37) since c = ∞ and 0 < ρ < 1

= C(s, sρ)e−sµt∞∑

n=0

(sµρt)n

n!rearranging terms

= C(s, sρ)e−sµtesµρt since∞∑

n=0

an

n!= ea

= C(s, sρ)e−sµt+sµρt

= C(s, sρ)e−sµ(1−ρ)t.

ThusP(Wq > t) = C(s, sρ)e−sµ(1−ρ)t (3.38)

for t ≥ 0 and P(Wq > t) = 1 for t < 0. Moreover, (3.36) may be rewritten as

Wqd= [1− C(s, sρ)]O ⊕ C(s, sρ)Exp(sµ(1− ρ)). (3.39)

Note that this result could have been derived directly from (3.36) by noting that if c = ∞, thenmodG(c−1, 1−ρ) is the distribution modG(1−ρ) and so (N?

c−1 +1) ∼ G(1−ρ). Now, using (2.9),it follows that the random sum of a G(1 − ρ) number of independent and identically distributedrandom variables with exponential distribution with parameter sµ,

∑N?c−1+1

i=1 Ei has distributionExp(sµ(1− ρ)).

Using (3.25)-(3.26), (3.39), and the fact that E[O] = Var[O] = 0, we get that for the M/M/ssystem,

E[Wq] =C(s, sρ)sµ(1− ρ)

, Var[Wq] =C(s, sρ)[2− C(s, sρ)]

[sµ(1− ρ)]2. (3.40)

Note that, in particular, both the expected value and the variance of the steady-state number ofcustomers in the queue tend to infinity as ρ tends to 1. Similarly, using (3.24), (3.36) and Wald´sequation, we get that for the M/M/s/s + c system with finite queue capacity c > 1,

E[Wq] = C(s, sρ, c− 1)E[1 + N?c−1]E[E1]

=[

11− ρ

× 1− (c + 1)ρc + cρc+1

1− ρc

]C(s, sρ, c− 1)

sµ(3.41)

if ρ 6= 1, and E[Wq] = c+12sµ C(s, s, c− 1) if ρ = 1.

Example 44 For the M/M/1 system, as the delay probability is equal to the traffic intensity, i.e.,C(1, ρ) = ρ, (3.40) leads to

E[Wq] =ρ

µ(1− ρ), Var[Wq] =

ρ(2− ρ)[µ(1− ρ)]2

. (3.42)

Similarly, from Example 40, we have that the delay probability in the M/M/2 system isC(2, ρ) = 2ρ2/(1 + ρ). Thus, using (3.40),

E[Wq] =ρ2

µ(1− ρ2), Var[Wq] =

[ρ

µ(1− ρ2)

]2

[1 + ρ(1− ρ)] . (3.43)

86

Note that relating the formulas (3.42) and (3.43) for the steady-state expected waiting time inqueue in M/M/s systems with the corresponding formulas (3.32) and (3.33) for the steady-stateexpected number of customers in queue in both systems, we conclude that E[Lq] = λE[Wq]. Thatis, the steady-state expected waiting time in queue is equal to the input rate times the steady-stateexpected number of customers in queue. This is an instance of Little’s formula, which will bedescribed in the next section. ¥

Example 45 For the system M/M/1/c, with c ≥ 1, the loss-delay probability is

C(1, ρ, c− 1) =ρ

∑c−1k=0 ρk

1 + ρ∑c−1

k=0 ρk=

∑ck=1 ρk

∑ck=0 ρk

=

{ρ−ρc+1

1−ρc+1 ρ 6= 1c

c+1 ρ = 1.

Thus, in view of (3.41), the steady-state expected waiting time in queue in the M/M/1/1+c systemis

E[Wq] =[

11− ρ

× 1− (c + 1)ρc + cρc+1

1− ρc

]ρ− ρc+1

µ(1− ρc+1)

=ρ

µ(1− ρ)× 1− (c + 1)ρc + cρc+1

1− ρc+1(3.44)

if ρ 6= 1, and E[Wq] = c+12µ

cc+1 = c

2µ if ρ = 1.

For ρ 6= 1, since B(1, ρ, c) = ρc+1(1− ρ)/(1− ρc+2), it follows that

λeE[Wq] = λ[1−B(1, ρ, c]E[Wq]

= λ1− ρc+1

1− ρc+2× ρ

µ(1− ρ)× 1− (c + 1)ρc + cρc+1

1− ρc+1

=ρ2

(1− ρ)× 1− (c + 1)ρc + cρc+1

1− ρc+2

= E[Lq].

As B(1, 1, c) = 1/(c + 2), when ρ = 1,

λeE[Wq] = λ[1−B(1, 1, c]E[Wq] = λc + 1c + 2

c

2µ=

c(c + 1)2(c + 2)

= E[Lq].

Thus, E[Lq] = λeE[Wq] for the M/M/1/1 + c system for any traffic intensity ρ, or any arrival rateλ and service rate µ. This is another instance of Little’s formula and expresses the fact that thethe steady-state expected waiting time in queue is equal to the input rate times the steady-stateexpected number of customers in queue. ¥

3.7 Little’s formula

In this section we introduce Little´s formula and exemplify how it can be used to derive results forqueueing systems.

Consider a stable M/M/s/s + c system and the following two possible method of collectingrewards:

(1) Each customer pays continuously at a rate $ 1 per unit time of presence in the system; and

87

(2) Each customer pays when entering the system the value of its sojourn time in the system indollars.

If method (1) is used, the system collects reward at rate of j dollars when there are j customersin the system, and thus the long-run reward rate per unit time will be, in face of (2.63),

s+c∑

j=0

jpj = E[Ls]

where the probabilities pj are given by (3.6). Similarly, if method (2) is used, then the long-runreward rate per unit time will be the long-run rate at which customers enter the system, i.e., theinput rate λe = λ[1 − B(s, sρ, c)], times the steady-state expected sojourn time of a customer inthe system, E[Ws]. But since the system is stable, and therefore the rate at which customers enterthe system is equal to the rate at which customers leave the system, the two methods of collectingrewards should produce the same long-run reward rate per time unit, i.e.,

E[Ls] = λeE[Ws] = λ[1−B(s, sρ, c)]E[Ws]. (3.45)

This relation is known as Little’s formula; it says that for a stable system the long-run averagenumber of customers in the system is equal to the input rate times the long-run average sojourntime of customers in the system.

Little’s formula is valid for stable systems with arrival processes other than the Poisson process.If the reward collection methods are changed so that customers pay $ 1 only when they are in queue(in method (1)) and pay only their waiting time in queue in dollars (in method (2)), then we obtainsimilarly,

E[Lq] = λeE[Wq] = λ[1−B(s, sρ, c)]E[Wq]. (3.46)

This is also called Little’s formula (for the number of customers in queue and waiting time inqueue). The formula says that for a stable system the long-run average number of customers in thequeue is equal to the input rate times the long-run average waiting time of customers in queue.

Note also that for stable M/M/s systems, (3.45) and (3.46) are just

E[Ls] = λE[Ws], E[Lq] = λE[Wq] (3.47)

since the input rate λe is just the arrival rate λ. Note that since

Wsd= S + Wq (3.48)

where S ∼ Exp(µ) represents the service time of an entering customer and S and Wq are indepen-dent, we have

E[Ws] =1µ

+ E[Wq]. (3.49)

The previous equation along with Littles’s formula may be used to derive E[Wq], E[Ws] and E[Ls]from E[Ls], since

E[Wq] =E[Lq]

λeby (3.46)

E[Ws] =1µ

+ E[Wq] =1µ

+E[Lq]

λeby (3.49)

E[Ls] = λeE[Ws] =λe

µ+ E[Lq] = sρe + E[Lq] by (3.45).

88

Note that from the last relation we have

E[Ls − Lq] = ae = sρe = sρ[1−B(s, sρ, c]. (3.50)

As Ls − Lq is the steady-state number of busy servers, this relation says that the steady-stateexpected number of busy servers equals the effective load or s times the effective traffic intensity.If, in particular, c = ∞, then E[Ls − Lq] = sρ.

We next give the representations of the steady-state sojourn time and number of customers inthe system, Ws and Ls. Namely,

Wsd= [1− C(s, sρ, c− 1)]S ⊕ C(s, sρ, c− 1)

S +

N?c−1+1∑

i=1

Ei

(3.51)

where S ∼ Exp(sµ), Eiiid∼ Exp(sµ), and S and {Ei, i = 1, 2, . . .} are independent. Similarly,

Lsd= [1− C(s, sρ, c)](X|X < s)⊕ C(s, sρ, c)(s + N?

c ). (3.52)

where X ∼ P (sρ) and N?c ∼ modG(c, 1−ρ). This follows since the steady-state number of customers

in the system given that not all servers are busy has a truncated Poisson distribution. The relations(3.51)-(3.52) may be used to derive additional results for M/M/s/s + c systems. However, we willonly use them to derive additional results for the M/M/1 system.

3.7.1 M/M/1 system

The steady-state number of customers in the M/M/1 system has a modified geometric distributionwith parameter (1− ρ), Ls ∼ modG(1− ρ), i.e.,

P(Ls = k) = (1− ρ)ρk (3.53)

for k ∈ IN0, and, as a consequence,

E[Ls] =ρ

1− ρ, Var[Ls] =

ρ

(1− ρ)2. (3.54)

As, due to the PASTA property, the steady-state number of customers that an entering cus-tomers finds in the system is Ls ∼ modG(1−ρ) his sojourn time in the system Ws has the followingdistribution:

Wsd=

Ls+1∑

i=1

Eid= Exp(µ(1− ρ)) (3.55)

where Eiiid∼ Exp(sµ) being independent of Ls. The last equality follows from (2.9) since (Ls +1) ∼

G(1− ρ). Thus, the steady-state sojourn time in the M/M/1 system has exponential distributionwith parameter µ(1− ρ) = µ− λ, so that

E[Ws] =1

µ− λ, Var[Ws] =

1(µ− λ)2

. (3.56)

Note that the expected value and variance of the steady-state sojourn time of customers in theM/M/1 system increases when the arrival rate increases and approaches infinity when the arrivalrate tends to the service rate. Thus, large sojourn times are likely to occur when λ is close to µ.

89

3.8 M/M/∞ and M/G/∞ systems

In this section we consider systems with infinite number of servers. The M/G/∞ is appropriate tomodel, e.g., self-service systems having a Poisson customer arrival process with customers spendingindependent amounts of time in the system with common general distribution.

As before, we let Q(t) represent the number of customers in the system at time t, for t ≥ 0.We look first to the case where the service time distribution is exponential; in this case the systemobtained is the M/M/∞ system.

3.8.1 M/M/∞ system

In the M/M/∞ the process Q is a birth-death process with state space IN0, the arrival rate isconstant and equal to λ and the service rate is (kµ) when there are k customers in the system. Ifour definition of traffic intensity if used for the system it returns the value zero for any values ofλ and µ. This is due to the fact that, since the system has an infinite number of servers and thearrival rate λ is finite, the load per server is zero. Thus, it will not be strange to conclude that thesystem is stable, as we will shortly conclude. As λi/µi+1 = a/(i + 1), for i ∈ IN0,

i−1∏

k=0

λk

µk+1=

ai

i!

for i ∈ IN0. As a result,∞∑

i=0

i−1∏

k=0

λk

µk+1=

∞∑

i=0

ai

i!= ea

and, from (3.6), Q has stationary distribution

pi = e−a ai

i!(3.57)

for i ∈ IN0. Thus, the system is stable for any arrival and service rate and its steady-state distri-bution is Poisson with parameter a = λ/µ.

In the M/M/∞ there is no queue and therefore customers do not wait in queue. Moreover,the sojourn time of each customer in the system equals its service time and has, therefore, anexponential distribution with parameter µ. If, as before, we let Ls denote the steady-state numberof customers in the system and Ws denote the steady-state sojourn time of a customer in thesystem,

Ls ∼ P (λ/µ), Ws ∼ Exp(µ). (3.58)

3.8.2 M/G/∞ system

We consider now the M/G/∞ system with Poisson arrival process with rate λ and service timedistribution function G with finite expected value µ−1. That is, if we let Si denote the service timeof the i-th customer that arrives to the system, then {Si, i = 1, 2, . . .} ∼ G, where

G(t) = P(S1 ≤ t)

for t ∈ IR. The process Q is not a CTMC unless the service time distribution is exponential. Thus,to characterize the distribution of Q(t) we will use the following result for Poisson processes:

90

Let {N(t), t ≥ 0} ∼ PP (λ) and t > 0 be fixed. If an arrival that takes place at time s, 0 ≤ s ≤ t,is registered, independently of other arrivals, with probability p(s), then the number of arrivals

that are registered until time t, R, has Poisson distribution, namely

R ∼ P

(λ

∫ t

0p(s) ds

). (3.59)

We now apply the previous result to derive the distribution of the number of customers in theM/G/∞ system at time t, Q(t). We assume that the system is initially empty, i.e., Q(0) = 0. As acustomer that arrives to the system at time s, 0 ≤ s ≤ t, is still in the system at time t if its servicetime is larger than t−s, the probability that the customer is in the system at time t is 1−G(t−s).As the service times of different customers are independent random variables, the previous resultgives that

Q(t) ∼ P

(λ

∫ t

0[1−G(t− s)] ds

)

and making the change of variable x = t− s, we get

Q(t) ∼ P

(λ

∫ t

0[1−G(x)] dx

). (3.60)

Note that if, in particular, the service time is Exp(µ), then [1−G(x)] = e−µx and thus∫ t

0λ[1−G(x)] dx =

∫ t

0λe−µx dx =

λ

µ[1− e−µt].

Thus, in the M/M/∞ system

Q(t) ∼ P

(λ

µ[1− e−µt]

). (3.61)

Moreover, since1µ

=∫ ∞

0[1−G(u)] du

it follows that Q possesses a limit distribution given by[

limt→∞Q(t)

]∼ P (λ/µ) . (3.62)

Thus, when time tends to infinity, the number of customers in the M/G/∞ system tends to aPoisson random variable with parameter λ/µ. Note that λ/µ is the expected sum of the timesspent by the customers that, in the long-run, enter the system during one unit of time.

Example 46 Users arrive at a library according to a Poisson process of rate 3/m and spend in thelibrary an amount of time (m) with U([10, 210]) distribution. Then the process Q = {Q(t), t ≥ 0},where Q(t) is the number of users in the library at time t m is the number of customers in anM/G/∞ system with arrival rate λ and U([10, 210]) service time distribution.

From (3.60), the distribution of the number of users in the library two hours after the libraryopens is Poisson with parameter

3∫ 120

0

[1− 120− u

210− 10

]du = 360− 3

∫ 120

0

v

200dv = 360− 3

(v2

2× 200

]120

0

= 360− 3× 1202

400= 360− 108 = 252. ¥

91

3.9 Jackson networks

In this section we consider queueing networks with multiple stations of M/M/s type. For thesection, it is important to recall that the departure process from a stable M/M/s system in steadystate is a Poisson process (with the same rate as the arrival process) and that the CTMC associatedto the number of customers in the system is time-reversible.

Example 47 Suppose that items arrive according to a Poisson process of rate 2/m to be processedin order by stations 1 and 2. Both stations have infinite capacity and identical servers at eachstation; the first station has one server that takes an exponential time with rate µ1 = 3 to processan item, and the second station has two servers that take an exponential time with rate µ2 = 5/4to process an item.

Note that if for i = 1, 2 and t ≥ 0 we let

Qi(t) = number of items in station i at time t

then (Q1, Q2) = {(Q1(t), Q2(t)), t ≥ 0} is a CTMC with state space IN20 , whose transition rates

are not difficult to derive if we are interested in transient quantities associated to the CTMC. Herehowever, we will be interested in deciding if the system is stable and, if yes, to compute the steadystate distribution of the number of customers in stations 1 and 2, which we denote by Ls1 and Ls2

whenever they exist.As the first station is an M/M/1 station with arrival rate λ1 = 2 and service rate µ1 = 3 > 2 = λ,

Q1 is a stable M/M/1 queue with traffic intensity ρ1 = λ1/µ1 = 2/3. In steady state the departureprocess from station 1, which is the arrival process to station 2 is a Poisson process with rateλ2 = 2. As the two servers at station 2 have service rate µ2 = 5/4, the traffic intensity at station2 is ρ2 = λ2/(2µ2) = 4/5 < 1, Q2 is a stable M/M/2 queue.

The ideas set forth above could be made rigorous to conclude that in steady-state (Q1, Q2) is atime-reversible CTMC whose steady state distribution is the product of the steady-state distributionof the number of customers in M/M/1 and M/M/2 systems with traffic intensities 2/3 and 4/5,respectively. Using (3.53), we have

P(Ls1 = k1) = ρk11 (1− ρ1) =

(23

)k1 13

for k1 ∈ IN0. Moreover, in view of (3.6)

P(Ls2 = k2) =

{1−ρ2

1+ρ2(2ρ2)k2 k2 = 0, 1

1−ρ2

1+ρ22ρk2

2 k = 2, 3, . . .=

{(1/9)(8/5)k2 k2 = 0, 1(2/9)(4/5)k2 k = 2, 3, . . .

.

Thus, the steady-state joint distribution of the number of items at stations 1 and 2 is

P(Ls1 = k1, Ls2 = k2) = P(Ls1 = k1)×P(Ls2 = k2)

=

{(1/27)(2/3)k1(8/5)k2 k1 ∈ IN0, k2 = 0, 1(2/27)(2/3)k1(4/5)k2 k1 ∈ IN0, k2 = 2, 3, . . .

. (3.63)

Moreover, taking into account (3.54), (3.33) and (3.50), the steady-state expected total number ofitems in the system is

E[Ls1 + Ls2] = E[Ls1] + E[Ls2] =ρ1

1− ρ1+

[2ρ2 +

2ρ32

1− ρ22

]= 2 +

85

+12845

=589

. ¥

92

The previous example is a particular case of a (tandem) feed forward network, in which theoutput from a station may only exit the network or be directed to stations with higher indices.Thus, no customer is serviced more than once at at each station. In the next example we willconsidered a variant of the previous example where a customer may be serviced more than once atthe second station.

Example 48 Consider the setting described in Example 47 and assume that everything remainsthe same with the exception that a fraction p of the items need reprocessing at station 2 aftercompleting the process. More precisely, after being processed at station 2 each item needs, inde-pendently of other items, to be reprocessed at station 2 with probability p (with this happeningafter each processing at station two regardless of the number of times the item has been processedat station 2).

Under these conditions, the number of times that an item requests processing at station 2is a geometric random variable with parameter p and, thus, has expected value 1/(1 − p). As,as concluded in Example 47, the departure process from station 1, which is the arrival processto station 2 from station 1, is a Poisson process with rate 2. Now, as if the M/M/2 systemcorresponding to station 2 is stable then each item will requests processing at station 2 a numberof times with geometric distribution with parameter p, if the system is stable, then the effectivearrival rate to station 2, λe2 will be given by

λe2 =2

1− p. (3.64)

Thus, the system is stable if and only if λe2 < 2µ2 or

21− p

< 2× 54⇐⇒ 1

1− p<

54⇐⇒ 1− p >

45⇐⇒ p <

15. (3.65)

Note that the effective arrival rate to station 2 is λe2 = λ[1 − p]−1 and, thus, it may besubstantially larger than the external arrival rate λ if the reprocessing probability p is not close tozero. If, e.g., p = 1/10, then the system is stable, λe2 = 2[1 − 1/10]−1 = 20/9 and the effectivetraffic intensity at station 2 is ρe2 = 8/9. In this situation, and using the notation of Example 47,the steady-state (Q1, Q2) is a time-reversible CTMC whose steady state distribution is the productof the steady-state distribution of the number of customers in M/M/1 and M/M/2 systems withtraffic intensities 2/3 and 8/9, respectively. Thus, the steady-state joint distribution of the numberof items at stations 1 and 2 is

P(Ls1 = k1, Ls2 = k2) = P(Ls1 = k1)×P(Ls2 = k2)

=

{(1/51)(2/3)k1(16/9)k2 k1 ∈ IN0, k2 = 0, 1(2/51)(2/3)k1(8/9)k2 k1 ∈ IN0, k2 = 2, 3, . . .

. ¥

Moreover, the steady-state expected total number of items in the system is

E[Ls1 + Ls2] = E[Ls1] + E[Ls2] =ρ1

1− ρ1+

[2ρ2 +

2ρ32

1− ρ22

]= 2 +

169

+1024153

=17817

. ¥

We can now define what is a Jackson network, which constitutes a generalization of the typeof network treated in Example 48 in terms of the number of stations and the type of movementsmade by customers among the stations of the network.

A network with N stations, numbered 1, 2, . . . , N , is a Jackson network if:

93

· The external arrival processes to stations 1, 2, . . . , N are independent Poisson processes, withthe arrival process to station i having rate λi, for i = 1, 2, . . . , N ;

· Station i, i = 1, 2, . . . , N , has infinite capacity, si identical servers, and the processing timesof customers at the servers of station i are independent random variables with Exp(µi) dis-tribution;

· After being served at station i each customer, independently of other customers or otherchoices made by the customer itself, seeks service at station j (may be equal to i) withprobability pij or otherwise leaves the system with probability 1−∑N

j=1 pij .

Now, let λei denote the effective arrival rate to station i, supposing that the system is stable. Bya reasoning similar to the one done to compute the expected number of steps needed for a DTMCto visit a state (here we are interested in a customer leaving the network) we would conclude that

λej = λj +N∑

i=1

λeipij (3.66)

for j = 1, 2, . . . , N , as if the system is stable the effective arrival rate to station i, λei, would haveto be equal to the output rate from the station, for each i. This would imply that the rate at whichcustomers departing from station i would seek service at station j would be equal to λeipij . As inaddition, station j has external arrival rate λj the previous equation follows.

Let the row vectors λ =[λ1 λ2 . . . λN

]and λe =

[λe1 λe2 . . . λeN

]denote the vectors

of external (effective) arrival rates to the stations of the network. Moreover, let P = [pij ]i,j=1,2,...,N

denote the routing probabilities of customers among stations after being served at a station. Then,the system (3.66) may be rewritten as λe = λ + λeP , which has a solution if I − P is invertible, inwhich case

λe = λ[I − P ]−1. (3.67)

The condition of I − P being invertible is equivalent to the expected number of times a customerseeks service at station j being finite, independently of the station at which the customer initiallyenters the network, for any station j. This is still the same as the expected total number of servicesseek by a customer at all stations being finite. From now onwards, we will assume that I − P isinvertible.

We let Qi(t) denote the number of customers at station i at time t, for i = 1, 2, . . . , N and t ≥ 0,and let Q = (Q1, Q2, . . . , QN ) = {(Q1(t), Q2(t), . . . , QN (t)), t ≥ 0}. The process Q is an irreducibleCTMC with state space INN

0 which has stationary distribution (i.e., the Jackson network is stable)if and only if

ρi =λei

sµi< 1, i = 1, 2, . . . , N (3.68)

in which case, the steady-state distribution of Q is the independent product of the steady-statedistributions of M/M/si systems with (effective) traffic intensity ρi, i = 1, 2, . . . , N . Note that wehave dropped the “e” in the notation for the effective traffic intensity to simplify the notation.

We let (Ls1, Ls2, . . . , LsN ) denote a random vector with distribution equal to the steady-statedistribution of Q, so that limt→∞Q(t) d= (Ls1, Ls2, . . . , LsN ). The random variable Lsi denotes thesteady-state number of customers at station i, for i = 1, 2, . . . , N . As the steady-state number ofcustomers at the various stations are independent random variables, it follows that

P(Ls1 = k1, Ls2 = k2, . . . , LsN = kN ) =N∏

l=1

P(Lsl = kl) (3.69)

94

for k1, k2, . . . , kN ∈ IN0 and moreover, in view of (3.6),

P(Lsl = kl) =

(slρl)i

i!Ps−1j=0

(slρl)j

j!+

(slρl)s

s!(1−ρl)

i = 0, 1, . . . , sl − 1

(slρl)s

s!ρi−s

lPs−1j=0

(slρl)j

j!+

(slρl)s

s!(1−ρl)

i = sl, sl + 1, . . .

. (3.70)

Thus all the steady-state analysis that has been developed for M/M/s single stations may beapplied to characterize each of the stations of Jackson networks

The properties (3.69)-(3.70) may be used to derive steady-state quantities of Jackson networks.For example, the steady-state expected number of customers at station i is

E[Lsi] = siρi +ρi

1− ρiC(si, siρi)

and the steady-state expected number of customers in the network is

N∑

i=1

E[Lsi] =N∑

i=1

siρi +N∑

i=1

ρi

1− ρiC(si, siρi).

The first (second) sum in the previous formula is the expected number of busy servers (customersin queue) in the entire network.

Note that in a stable Jackson network the input rate to the (entire) network equals the outputrate from the network, so that

N∑

i=1

λi =N∑

i=1

λei

1−

N∑

j=1

pij

(3.71)

or, in matrix notation,λ1 = λe[I − P ]1. (3.72)

This relation follows directly from (3.66) by adding up all the equations.

3.10 Final remarks

In this chapter we have studied some queueing networks which may be modelled as CTMCs andintroduced some important parameters and performance measures of queueing networks. Thereare other queueing networks that may be studied totally or partially using DTMCs and CTMCs;these include:

· Closed Jackson networks: these networks are like the open Jackson networks with the excep-tion that there are no external arrivals and no customers depart from the system;

· G/M/s/s + c systems: the state of the system at the times of arrivals of customers to thesystem is a DTMC;

· M/G/1 systems: the state of the system after departures of customers from the system is aDTMC – see Problem 2 of Test # 1.

95