4
$\begingroup$

Why is mutual information symmetric, meaning why does I(A,B) = I(B,A)? Isnt the definition of mutual information, I(A,B), something like "the reduction of entropy in A when given B"? P(A|B) doesnt equal P(B|A) right?

$\endgroup$
3
$\begingroup$

Given the definition for mutual information

$$I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) },$$

it follows from rearrangement of the summands

$$I(Y; X) = \sum_{x \in X} \sum_{y \in Y} p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) }.$$

Hence $I(X; Y) = I(Y; X)$.


Edit

"$I(X;Y)$ measures the average reduction in uncertainty of $X$ that results from knowing $Y$"

If you interpret $H(X) - H(X|Y)$ where $H(X)$ is the marginal entropy of $X$ and $H(X|Y)$ the conditional entropy of $X$ given $Y$ as the reduction in uncertainty, then it is equal to $H(Y) - H(Y|X)$ and $I(X; Y)$ (see Wikipedia).

$\endgroup$
  • $\begingroup$ but intuitively how does it make sense for it to be symmetric? $\endgroup$ – Armon Safai Jun 25 '16 at 17:51
  • $\begingroup$ According to the definition mutual information is a measure of how the joint distribution of two random variables deviates from the case where they are independent from each other. For the joint distribution there's no asymmetry, so mutual information is symmetric. $\endgroup$ – aventurin Jun 25 '16 at 18:29
  • $\begingroup$ @aventurin But i was told from many sources that mutual information is something like "I(X;Y) measures the average reduction in uncertainty of X that results from knowing Y" $\endgroup$ – Armon Safai Jun 25 '16 at 19:01
  • $\begingroup$ I have extended my answer. Does this help? $\endgroup$ – aventurin Jun 25 '16 at 21:16
  • $\begingroup$ Well back to my question: intuitively how does it make sense for it to be symmetric? given what we just said is the definition? $\endgroup$ – Armon Safai Jun 26 '16 at 2:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.