Template:Short description Template:Information theory
In information theory, Shannon's source coding theorem (or noiseless coding theorem) establishes the statistical limits to possible data compression for data whose source is an independent identically-distributed random variable, and the operational meaning of the Shannon entropy.
Named after Claude Shannon, the source coding theorem shows that, in the limit, as the length of a stream of independent and identically-distributed random variable (i.i.d.) data tends to infinity, it is impossible to compress such data such that the code rate (average number of bits per symbol) is less than the Shannon entropy of the source, without it being virtually certain that information will be lost. However it is possible to get the code rate arbitrarily close to the Shannon entropy, with negligible probability of loss.
The source coding theorem for symbol codes places an upper and a lower bound on the minimal possible expected length of codewords as a function of the entropy of the input word (which is viewed as a random variable) and of the size of the target alphabet.
Note that, for data that exhibits more dependencies (whose source is not an i.i.d. random variable), the Kolmogorov complexity, which quantifies the minimal description length of an object, is more suitable to describe the limits of data compression. Shannon entropy takes into account only frequency regularities while Kolmogorov complexity takes into account all algorithmic regularities, so in general the latter is smaller. On the other hand, if an object is generated by a random process in such a way that it has only frequency regularities, entropy is close to complexity with high probability (Shen et al. 2017).<ref name="Shen2017"/>
StatementsEdit
Source coding is a mapping from (a sequence of) symbols from an information source to a sequence of alphabet symbols (usually bits) such that the source symbols can be exactly recovered from the binary bits (lossless source coding) or recovered within some distortion (lossy source coding). This is one approach to data compression.
Source coding theoremEdit
In information theory, the source coding theorem (Shannon 1948)<ref name="Shannon"/> informally states that (MacKay 2003, pg. 81,<ref name="MacKay"/> Cover 2006, Chapter 5<ref name="Cover"/>):
Template:Mvar i.i.d. random variables each with entropy Template:Math can be compressed into more than Template:Math bits with negligible risk of information loss, as Template:Math; but conversely, if they are compressed into fewer than Template:Math bits it is virtually certain that information will be lost.
The <math>NH(X)</math> coded sequence represents the compressed message in a biunivocal way, under the assumption that the decoder knows the source. From a practical point of view, this hypothesis is not always true. Consequently, when the entropy encoding is applied the transmitted message is <math>NH(X)+(inf. source)</math>. Usually, the information that characterizes the source is inserted at the beginning of the transmitted message.
Source coding theorem for symbol codesEdit
Let Template:Math denote two finite alphabets and let Template:Math and Template:Math denote the set of all finite words from those alphabets (respectively).
Suppose that Template:Mvar is a random variable taking values in Template:Math and let Template:Math be a uniquely decodable code from Template:Math to Template:Math where Template:Math. Let Template:Mvar denote the random variable given by the length of codeword Template:Math.
If Template:Math is optimal in the sense that it has the minimal expected word length for Template:Mvar, then (Shannon 1948):
- <math> \frac{H(X)}{\log_2 a} \leq \mathbb{E}[S] < \frac{H(X)}{\log_2 a} +1 </math>
Where <math>\mathbb{E}</math> denotes the expected value operator.
Proof: source coding theoremEdit
Given Template:Mvar is an i.i.d. source, its time series Template:Math is i.i.d. with entropy Template:Math in the discrete-valued case and differential entropy in the continuous-valued case. The Source coding theorem states that for any Template:Math, i.e. for any rate Template:Math larger than the entropy of the source, there is large enough Template:Mvar and an encoder that takes Template:Mvar i.i.d. repetition of the source, Template:Math, and maps it to Template:Math binary bits such that the source symbols Template:Math are recoverable from the binary bits with probability of at least Template:Math.
Proof of Achievability. Fix some Template:Math, and let
- <math>p(x_1, \ldots, x_n) = \Pr \left[X_1 = x_1, \cdots, X_n = x_n \right].</math>
The typical set, Template:Math, is defined as follows:
- <math>A_n^\varepsilon =\left\{(x_1, \cdots, x_n) \ : \ \left|-\frac{1}{n} \log p(x_1, \cdots, x_n) - H_n(X)\right| < \varepsilon \right\}.</math>
The asymptotic equipartition property (AEP) shows that for large enough Template:Mvar, the probability that a sequence generated by the source lies in the typical set, Template:Math, as defined approaches one. In particular, for sufficiently large Template:Mvar, <math>P((X_1,X_2,\cdots,X_n) \in A_n^\varepsilon)</math> can be made arbitrarily close to 1, and specifically, greater than <math>1-\varepsilon</math> (See AEP for a proof).
The definition of typical sets implies that those sequences that lie in the typical set satisfy:
- <math>2^{-n(H(X)+\varepsilon)} \leq p \left (x_1, \cdots, x_n \right ) \leq 2^{-n(H(X)-\varepsilon)}</math>
- The probability of a sequence <math>(X_1,X_2,\cdots X_n)</math> being drawn from Template:Math is greater than Template:Math.
- <math>\left| A_n^\varepsilon \right| \leq 2^{n(H(X)+\varepsilon)}</math>, which follows from the left hand side (lower bound) for <math> p(x_1,x_2,\cdots x_n)</math>.
- <math>\left| A_n^\varepsilon \right| \geq (1-\varepsilon) 2^{n(H(X)-\varepsilon)}</math>, which follows from upper bound for <math> p(x_1,x_2,\cdots x_n)</math> and the lower bound on the total probability of the whole set Template:Math.
Since <math>\left| A_n^\varepsilon \right| \leq 2^{n(H(X)+\varepsilon)}, n(H(X)+\varepsilon)</math> bits are enough to point to any string in this set.
The encoding algorithm: the encoder checks if the input sequence lies within the typical set; if yes, it outputs the index of the input sequence within the typical set; if not, the encoder outputs an arbitrary Template:Math digit number. As long as the input sequence lies within the typical set (with probability at least Template:Math), the encoder does not make any error. So, the probability of error of the encoder is bounded above by Template:Mvar.
Proof of converse: the converse is proved by showing that any set of size smaller than Template:Math (in the sense of exponent) would cover a set of probability bounded away from Template:Math.
Proof: Source coding theorem for symbol codesEdit
For Template:Math let Template:Math denote the word length of each possible Template:Math. Define <math>q_i = a^{-s_i}/C</math>, where Template:Mvar is chosen so that Template:Math. Then
- <math>\begin{align}
H(X) &= -\sum_{i=1}^n p_i \log_2 p_i \\
&\leq -\sum_{i=1}^n p_i \log_2 q_i \\
&= -\sum_{i=1}^n p_i \log_2 a^{-s_i} + \sum_{i=1}^n p_i \log_2 C \\
&= -\sum_{i=1}^n p_i \log_2 a^{-s_i} + \log_2 C \\
&\leq -\sum_{i=1}^n - s_i p_i \log_2 a \\
&= \mathbb{E} S \log_2 a \\
\end{align}</math>
where the second line follows from Gibbs' inequality and the fifth line follows from Kraft's inequality:
- <math>C = \sum_{i=1}^n a^{-s_i} \leq 1</math>
so Template:Math.
For the second inequality we may set
- <math>s_i = \lceil - \log_a p_i \rceil </math>
so that
- <math> - \log_a p_i \leq s_i < -\log_a p_i + 1 </math>
and so
- <math> a^{-s_i} \leq p_i</math>
and
- <math> \sum a^{-s_i} \leq \sum p_i = 1</math>
and so by Kraft's inequality there exists a prefix-free code having those word lengths. Thus the minimal Template:Mvar satisfies
- <math>\begin{align}
\mathbb{E} S & = \sum p_i s_i \\ & < \sum p_i \left( -\log_a p_i +1 \right) \\ & = \sum - p_i \frac{\log_2 p_i}{\log_2 a} +1 \\ & = \frac{H(X)}{\log_2 a} +1 \\ \end{align}</math>
Extension to non-stationary independent sourcesEdit
Fixed rate lossless source coding for discrete time non-stationary independent sourcesEdit
Define typical set Template:Math as:
- <math>A_n^\varepsilon = \left \{x_1^n \ : \ \left|-\frac{1}{n} \log p \left (X_1, \cdots, X_n \right ) - \overline{H_n}(X)\right| < \varepsilon \right \}.</math>
Then, for given Template:Math, for Template:Mvar large enough, Template:Math. Now we just encode the sequences in the typical set, and usual methods in source coding show that the cardinality of this set is smaller than <math>2^{n(\overline{H_n}(X)+\varepsilon)}</math>. Thus, on an average, Template:Math bits suffice for encoding with probability greater than Template:Math, where Template:Mvar and Template:Mvar can be made arbitrarily small, by making Template:Mvar larger.