I want to understanding the meaning of the difference of two information entropy values.

I have the following scenario. Let $x$ be a number of hours a user spend on some video sharing websites. Thus, we may have the sets:

$X_{A} = \{x_1,x_2,\cdots,x_{n_A}\}$, and $X_{B} = \{x_1,x_2,\cdots,x_{n_B}\}$ that represent the number of hours the users of $A$ and $B$ spent on the websites $A$ and $B$, respectively.

Now, we can calculate the Cumulative distribution function (CDF) probability values, described here, as follows: For a real-valued random variable X, the CDF is defined as:

$$F_A(x) = P(X\leq x),\ \forall x \in X_A$$

Similarly, we find $F_B(x)$.

Then we calculate the entropy values $E_A$ and $E_B$ for the probabilities $F_A(x)$ and $F_B(x)$ obtained above (as described here):

$$E_A = - \sum\limits_{i=1}^{n_A}P(x_i)log(P(x_i))$$

Similarly, we find $E_B$.

Now my question is, what do $E_A$ and $E_B$ mean in this scenario? and what their difference also means?

  • 1
    $\begingroup$ Entropy is usually denoted by the symbol H. It quantifies the estimated information content of a signal; the higher it is, the more informative the signal is. Furthermore, information is related to predictability; a predictable signal is said to be uninformative and have low entropy. $H(A) - H(B)$ does not mean much but $H(A,B) - H(A) \equiv H(A|B)$ and $H(A,B) - H(B) \equiv H(B|A)$. Your detour into the CDF was unnecessary, by the way. I suggest reading the Wikipedia article. $\endgroup$
    – Emre
    May 15, 2017 at 18:04
  • 1
    $\begingroup$ related: stats.stackexchange.com/questions/246239/… $\endgroup$
    – Taylor
    May 16, 2017 at 4:34
  • $\begingroup$ Excuse me, I got the definitions of the conditional entropies reversed for $H(A|B)$ and $H(B|A)$. $\endgroup$
    – Emre
    May 16, 2017 at 18:39


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.