Shannon entropy
The measure of a random variable's uncertainty: how many bits it takes on average.
Shannon entropy measures the average information produced by a random source. It is an ensemble measure: it applies to a distribution, not to a single string. For a single string you need Kolmogorov complexity, which is a different object and is not computable.
Shannon does not ask messages to mean anything. The theory is purely syntactic, which is both its strength and its limit: it holds for any source, and it cannot see the computational structure generating the data.
The 1948 axiomatic characterisation is the part that gets forgotten and matters most: H is the only function that is continuous, maximal on the uniform distribution, and additive in the sense that H(X,Y) = H(X) + H(Y|X). Those three demands force the logarithmic form. It is not a definition picked for convenience.
Everything else follows: mutual information I(X;Y) as the reduction of uncertainty about X once Y is known, always non-negative; the Kullback-Leibler divergence as the canonical measure of how far one distribution sits from another, zero only when they coincide, and asymmetric — which makes it a useful measure rather than a distance.
Sources
- Shannon (1948)
- Cover & Thomas, Elements of Information Theory
- Jaynes, Probability Theory