Markov Chains

Classification of States

Johar AshfaqueA state j is said to be accessible from state i if for some n, the n-step transition probability is positivethat is to say there is positive probability of reaching state j starting from state i after some number oftime periods.

State i is called recurrent if for every state j accessible from state i, state i is also accessible from statej. In other words, let A(i) be the set of states that are accessible from state i. Then state i is recurrentif for all states j that belong to A(i) we have that the states i belong to A(j).

A state is called transient if it is not recurrent.

For state i being a recurrent state, the set of states A(i) forms a recurrent class.

• A Markov chain can be decomposed into one or more recurrent classes plus some possible transientstates.

• A recurrent state is accessible from all states in its class but is not accessible from recurrent statesin other classes.

• A transient state is not accessible from any recurrent state.

• At least one, possibly more, recurrent states are accessible from a given transient state.

1. We see that once a state enters a class of recurrent states it stays within the class; since all statesin the class are accessible from each other, all states in the class will be visited an infinite numberof times.

2. If the initial state is transient, then the initial portion consists of transient states and final portionwill consist of recurrent states from the same class.

Consider a recurrent class, denote it by R.

• The class is called periodic if its states can be grouped in d > 1 disjoint subsets S1, ..., Sd so thatall transitions from one subset lead to the next subset. More precisely, if

i ∈ Sk, and pij > 0 then

{j ∈ Sk+1 if k = 1, ..., d− 1

j ∈ S1 if k = d

• The class is aperiodic if and only if there exists a n and a state s in the class such that

pis(n) > 0

for all i ∈ R.

1