Intuition behind the entropy definition

The definition of information entropy is defined below:

This looks fine but I got no intuition why it is defined this way. Could any one share their ideas on this? Thanks!

• If you've time, pls. refer section 1.6 "Information Theory" from PRML, Bishop. (I need to revise, and may post an answer.) Jul 30 '19 at 3:37

Pls. refer Pattern recognition and Machine Learning for details - section 1.6: Information Theory.

Left graph is Information(events) vs Probability. And the right graph is Expected Information/(Uncertainity) vs Probability.

If we look carefully at the left graph; the variation at either extremes of the curve dies down, representing less change. Translated in current context, it means not much Information flows-in at extremes.

Staying with the left curve, it's somewhere in the middle (45-degree, from vertex) where there's relatively more "meat" in the sense that more change happens/ translated, more information flows-in.

Same thing is shown by the right curve: at two extremes, Change/ Expected Information is low at extremes, and highest at centre.

A crude Graph using Excel:

• so from the left figure, the bigger probability actually has smaller h(x) , i.e. fewer information. How do I perceive that higher probability is associated with fewer information? Jul 30 '19 at 18:04
• My response was too long. Pls. refer the (edited) Answer for response. Jul 31 '19 at 2:29
• Equation: p log(p) for all p in [0, 1] interval; it'll be zero if either p=0 or p=1, and maximum with p=1/2 i.e. (1/2)* log(1/2) Jul 31 '19 at 4:22
• Thanks for the answer. After reading your post, now my question becomes why would small probability correspond to high information? Any ideas? Aug 1 '19 at 5:42
• Information is degree of surprise and we're more surprised on learning about events that're highly improbable. And so, Lower probability (values) of event's occurrence maps to higher information (values) and vice-versa. Regards Aug 1 '19 at 7:33

You can think of it as a tree node of variable X with n branches where each branch has a depth relative to it's probability. The more balanced your tree at node X is, the higher it's entropy will be at X.

For n=2 the highest entropy is if the probability of each of the two branches is 0.5: