Editing Entropy (information theory) (section)

==Efficiency (normalized entropy)==
A source set <math>\mathcal{X}</math> with a non-uniform distribution will have less entropy than the same set with a uniform distribution (i.e. the "optimized alphabet"). This deficiency in entropy can be expressed as a ratio called efficiency:<ref>Indices of Qualitative Variation.
AR Wilcox - 1967 https://www.osti.gov/servlets/purl/4167340</ref>

<math display="block">\eta(X) = \frac{H}{H_\text{max}} = -\sum_{i=1}^n \frac{p(x_i) \log_b (p(x_i))}{\log_b (n)}.
</math>
Applying the basic properties of the logarithm, this quantity can also be expressed as:
<math display="block">\begin{align}
\eta(X) &= -\sum_{i=1}^n \frac{p(x_i) \log_b(p(x_i))}{\log_b (n)}
= \sum_{i=1}^n \frac{\log_b\left(p(x_i)^{-p(x_i)}\right)}{\log_b(n)} \\[1ex]
&= \sum_{i=1}^n \log_n\left(p(x_i)^{-p(x_i)}\right)
= \log_n \left(\prod_{i=1}^n p(x_i)^{-p(x_i)}\right).
\end{align}
</math>

Efficiency has utility in quantifying the effective use of a [[communication channel]]. This formulation is also referred to as the normalized entropy, as the entropy is divided by the maximum entropy <math>{\log_b (n)}</math>.  Furthermore, the efficiency is indifferent to the choice of (positive) base {{math|''b''}}, as indicated by the insensitivity within the final logarithm above thereto.