Question

You have an urn with 3 balls, some are red and some are blue. Let Bn, n = 0, 1, 2, . . ., denote the number of blue balls at time n. (Then the number of red balls at time n is 3 − Bn.) When we make a transition from time n to time n + 1, we pick one of the balls uniformly at random (i.e., pick each ball with equal probability), and then do one of the following

(a) with probability 1/4 leave this ball in the urn;

(b) with probability 3/4 replace this ball by a ball of opposite color.

Is {Bn, n ≥ 0} is a Markov Chain? If so, what is state space and transition probabilities?

Answer 1

{Bn, n ≥ 0} is a Markov Chain because the transition probabilities from state n to state n+1 will only depend on the current state (number of blue balls at time n). It will not depend on previous states as how the current state was achieved.

The number of blue balls at time n will be 0, 1, 2, 3. So, the state space is {0, 1, 2, 3}

For state 0, all balls will be red. So, the transition probability to state 0 is 1/4 and the transition probability to state 1 is 3/4

For state 1, the probability to pick red ball is 2/3 and probability to pick blue ball is 1/3.

Case 1.1 - When red ball is picked, the transition probability to state 1 is (2/3) * (1/4) = 1/6 and the transition probability to state 2 is (2/3) * (3/4) = 1/2

Case 1.2 - When blue ball is picked, the transition probability to state 1 is (1/3) * (1/4) = 1/12 and the transition probability to state 0 is (1/3) * (3/4) = 1/4

So,  the transition probability to state 0 is 1/4, to state 1 is (1/6) + (1/12) = 1/4 and to state 2 is 1/2

For state 2, the probability to pick red ball is 1/3 and probability to pick blue ball is 2/3.

Case 2.1 - When red ball is picked, the transition probability to state 2 is (1/3) * (1/4) = 1/12 and the transition probability to state 3 is (1/3) * (3/4) = 1/4

Case 2.2 - When blue ball is picked, the transition probability to state 2 is (2/3) * (1/4) = 1/6 and the transition probability to state 1 is (2/3) * (3/4) = 1/2

So,  the transition probability to state 1 is 1/2, to state 2 is (1/6) + (1/12) = 1/4 and to state 3 is 1/4

For state 3, all balls will be blue. So, the transition probability to state 3 is 1/4 and the transition probability to state 2 is 3/4

The transition probabilities matrix is,