Why is mutual information symmetric, meaning why does I(A,B) = I(B,A)? Isnt the definition of mutual information, I(A,B), something like "the reduction of entropy in A when given B"? P(A|B) doesnt equal P(B|A) right?


Given the definition for mutual information

$$I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) },$$

it follows from rearrangement of the summands

$$I(Y; X) = \sum_{x \in X} \sum_{y \in Y} p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) }.$$

Hence $I(X; Y) = I(Y; X)$.


"$I(X;Y)$ measures the average reduction in uncertainty of $X$ that results from knowing $Y$"

If you interpret $H(X) - H(X|Y)$ where $H(X)$ is the marginal entropy of $X$ and $H(X|Y)$ the conditional entropy of $X$ given $Y$ as the reduction in uncertainty, then it is equal to $H(Y) - H(Y|X)$ and $I(X; Y)$ (see Wikipedia).

| improve this answer | |
  • $\begingroup$ but intuitively how does it make sense for it to be symmetric? $\endgroup$ – Armon Safai Jun 25 '16 at 17:51
  • 1
    $\begingroup$ According to the definition mutual information is a measure of how the joint distribution of two random variables deviates from the case where they are independent from each other. For the joint distribution there's no asymmetry, so mutual information is symmetric. $\endgroup$ – aventurin Jun 25 '16 at 18:29
  • $\begingroup$ @aventurin But i was told from many sources that mutual information is something like "I(X;Y) measures the average reduction in uncertainty of X that results from knowing Y" $\endgroup$ – Armon Safai Jun 25 '16 at 19:01
  • $\begingroup$ I have extended my answer. Does this help? $\endgroup$ – aventurin Jun 25 '16 at 21:16
  • 1
    $\begingroup$ @Armon Safai: Define what you mean with intuition. Mutual information is defined to be symmetric. BTW, what is counter-intuitive when we say that the entropy of X given Y is equal to the entropy of X minus the mutual information of X and Y; and the entropy of Y given X is equal to the entropy of Y minus the mutual information of X and Y? (see my edit above) $\endgroup$ – aventurin Jun 29 '16 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.