Question

Describe the steady-state solution of Markov chains and the state probabilities:

(Typed answer only .no hand written):

Answer 1

Let us consider the following:
Sequence X₀, X₁, X₂, X₃,.. X_t,.... form a Markov chain.

s₁, s₂ ,...., s_k be the k states associated with the above Markov chain.

Also let the state transition matrix be T.

Here we consider the case where the transition probabilities are independent of time t.

X is known to have stationary transition probabilities.

The ith column of matrix T gives us the probabilities on the all the branches converging on to the ith state s_i.

The ith row of matrix T gives us the probabilities on the all the branches emerging from the ith state s_i.

Also let v_t be the probability distribution of X_t. v_t = (v_t[1], v_t[2], v_t[3], .., v_t[k]), where v_t[i] = P(X_t = s_i).

From law of total probability,

Thus we can write the following equation:

Suppose X₀ has some distribution v₀, then v₁=v₀T, v₂=v₀T², v₃=v₀T³, ..., v_n=v₀Tⁿ,..

The steady state distribution of the Markov chain will be:

Steady state Theorem:

If the Markov chain is irreducible and aperiodic, then irrespective of the initial distribution v₀, there exists an unique steady state distribution v^* such that v_t approaches v^*.

Solving the above linear equations gives us the steady state probabilities for the random variable X.

Some points to ponder:

The steady state distribution will be the Eigenvector for the transpose of the transition matrix with Eigenvalue = 1.

Now given that the process is converging to steady state, one can say that the probability of X being in some state i after n steps from now, does not depend on n.