I want to understanding the meaning of the difference of two information entropy values.

I have the following scenario. Let $x$ be a number of hours a user spend on some video sharing websites. Thus, we may have the sets:

$X_{A} = \{x_1,x_2,\cdots,x_{n_A}\}$, and $X_{B} = \{x_1,x_2,\cdots,x_{n_B}\}$ that represent the number of hours the users of $A$ and $B$ spent on the websites $A$ and $B$, respectively.

Now, we can calculate the Cumulative distribution function (CDF) probability values, described here, as follows: For a real-valued random variable X, the CDF is defined as:

$$F_A(x) = P(X\leq x),\ \forall x \in X_A$$

Similarly, we find $F_B(x)$.

Then we calculate the entropy values $E_A$ and $E_B$ for the probabilities $F_A(x)$ and $F_B(x)$ obtained above (as described here):

$$E_A = - \sum\limits_{i=1}^{n_A}P(x_i)log(P(x_i))$$

Similarly, we find $E_B$.

Now my question is, what do $E_A$ and $E_B$ mean in this scenario? and what their difference also means?

  • 1
    $\begingroup$ Entropy is usually denoted by the symbol H. It quantifies the estimated information content of a signal; the higher it is, the more informative the signal is. Furthermore, information is related to predictability; a predictable signal is said to be uninformative and have low entropy. $H(A) - H(B)$ does not mean much but $H(A,B) - H(A) \equiv H(A|B)$ and $H(A,B) - H(B) \equiv H(B|A)$. Your detour into the CDF was unnecessary, by the way. I suggest reading the Wikipedia article. $\endgroup$ – Emre May 15 '17 at 18:04
  • 1
    $\begingroup$ related: stats.stackexchange.com/questions/246239/… $\endgroup$ – Taylor May 16 '17 at 4:34
  • $\begingroup$ Excuse me, I got the definitions of the conditional entropies reversed for $H(A|B)$ and $H(B|A)$. $\endgroup$ – Emre May 16 '17 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.