Question

Consider a r.v. representing coin throws (Bernoulli Variable with Σ = {0,1} ). Let the true
probability distribution be p(0) = r, p(1) = 1-r.
Someone guesses a different distribution q(0) = s, q(1) = 1-s.
(a) Find expressions for the Kullback–Leibler distances D(p||q) and D(q||p) between the
two distributions in terms of r and s.
(b) Show that in general, D(p||q) ≠ D(q||p) and that equality occurs iff r = s.
(c) Compute D(p||q) and D(q||p) for the case r = 1/2 and s = 1/4.

Answer 1