1
$\begingroup$

The definition of information entropy is defined below:

enter image description here

This looks fine but I got no intuition why it is defined this way. Could any one share their ideas on this? Thanks!

$\endgroup$
  • $\begingroup$ If you've time, pls. refer section 1.6 "Information Theory" from PRML, Bishop. (I need to revise, and may post an answer.) $\endgroup$ – Continue2Learn Jul 30 '19 at 3:37
2
$\begingroup$

Pls. refer Pattern recognition and Machine Learning for details - section 1.6: Information Theory.

enter image description here

Left graph is Information(events) vs Probability. And the right graph is Expected Information/(Uncertainity) vs Probability.

If we look carefully at the left graph; the variation at either extremes of the curve dies down, representing less change. Translated in current context, it means not much Information flows-in at extremes.

Staying with the left curve, it's somewhere in the middle (45-degree, from vertex) where there's relatively more "meat" in the sense that more change happens/ translated, more information flows-in.

Same thing is shown by the right curve: at two extremes, Change/ Expected Information is low at extremes, and highest at centre.

A crude Graph using Excel:

enter image description here

| improve this answer | |
$\endgroup$
  • $\begingroup$ so from the left figure, the bigger probability actually has smaller h(x) , i.e. fewer information. How do I perceive that higher probability is associated with fewer information? $\endgroup$ – Edamame Jul 30 '19 at 18:04
  • $\begingroup$ My response was too long. Pls. refer the (edited) Answer for response. $\endgroup$ – Continue2Learn Jul 31 '19 at 2:29
  • $\begingroup$ Equation: p log(p) for all p in [0, 1] interval; it'll be zero if either p=0 or p=1, and maximum with p=1/2 i.e. (1/2)* log(1/2) $\endgroup$ – Continue2Learn Jul 31 '19 at 4:22
  • $\begingroup$ Thanks for the answer. After reading your post, now my question becomes why would small probability correspond to high information? Any ideas? $\endgroup$ – Edamame Aug 1 '19 at 5:42
  • 1
    $\begingroup$ Information is degree of surprise and we're more surprised on learning about events that're highly improbable. And so, Lower probability (values) of event's occurrence maps to higher information (values) and vice-versa. Regards $\endgroup$ – Continue2Learn Aug 1 '19 at 7:33
0
$\begingroup$

You can think of it as a tree node of variable X with n branches where each branch has a depth relative to it's probability. The more balanced your tree at node X is, the higher it's entropy will be at X.

For n=2 the highest entropy is if the probability of each of the two branches is 0.5: enter image description here

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.