←Research

Shannon entropy

The measure of a random variable's uncertainty: how many bits it takes on average.

Maximal for the uniform distribution, zero for a deterministic variable.

Shannon entropy measures the average information produced by a random source. It is an ensemble measure: it applies to a distribution, not to a single string. For a single string you need Kolmogorov complexity, which is a different object and is not computable.

Shannon does not ask messages to mean anything. The theory is purely syntactic, which is both its strength and its limit: it holds for any source, and it cannot see the computational structure generating the data.

The 1948 axiomatic characterisation is the part that gets forgotten and matters most: H is the only function that is continuous, maximal on the uniform distribution, and additive in the sense that H(X,Y) = H(X) + H(Y|X). Those three demands force the logarithmic form. It is not a definition picked for convenience.

Everything else follows: mutual information I(X;Y) as the reduction of uncertainty about X once Y is known, always non-negative; the Kullback-Leibler divergence as the canonical measure of how far one distribution sits from another, zero only when they coincide, and asymmetric — which makes it a useful measure rather than a distance.

Sources

  • Shannon (1948)
  • Cover & Thomas, Elements of Information Theory
  • Jaynes, Probability Theory