Editing Mutual information

{{Short description|Measure of dependence between two variables}}
{{CS1 config|mode=cs1}}
{{Information theory}}

[[File:Entropy-mutual-information-relative-entropy-relation-diagram.svg|thumb|256px|right|[[Venn diagram]] showing additive and subtractive relationships of various information measures associated with correlated variables <math>X</math> and <math>Y</math>.<ref>{{Cite book|last1=Cover|first1=Thomas M.|url=http://www.cs.columbia.edu/~vh/courses/LexicalSemantics/Association/Cover&Thomas-Ch2.pdf|title=Elements of information theory|last2=Thomas|first2=Joy A.|publisher=John Wiley & Sons, Ltd|year=2005|isbn=9780471748823|pages=13–55}}</ref> The area contained by either circle is the [[joint entropy]] <math>\Eta(X,Y)</math>. The circle on the left (red and violet) is the [[Entropy (information theory)|individual entropy]] <math>\Eta(X)</math>, with the red being the [[conditional entropy]] <math>\Eta(X\mid Y)</math>. The circle on the right (blue and violet) is <math>\Eta(Y)</math>, with the blue being <math>\Eta(Y\mid X)</math>. The violet is the mutual information <math>\operatorname{I}(X;Y)</math>.]]

In [[probability theory]] and [[information theory]], the '''mutual information''' ('''MI''') of two [[random variable]]s is a measure of the mutual [[Statistical dependence|dependence]] between the two variables. More specifically, it quantifies the "[[Information content|amount of information]]" (in [[Units of information|units]] such as [[shannon (unit)|shannon]]s ([[bit]]s), [[Nat (unit)|nats]] or [[Hartley (unit)|hartleys]]) obtained about one random variable by observing the other random variable. The concept of mutual information is intimately linked to that of [[Entropy (information theory)|entropy]] of a random variable, a fundamental notion in information theory that quantifies the expected "amount of information" held in a random variable.

Not limited to real-valued random variables and linear dependence like the [[Pearson correlation coefficient|correlation coefficient]], MI is more general and determines how different the [[joint distribution]] of the pair <math>(X,Y)</math> is from the product of the marginal distributions of <math>X</math> and <math>Y</math>. MI is the [[expected value]] of the [[pointwise mutual information]] (PMI).

The quantity was defined and analyzed by [[Claude Shannon]] in his landmark paper "[[A Mathematical Theory of Communication]]", although he did not call it "mutual information". This term was coined later by [[Robert Fano]].<ref>{{cite journal | last1 = Kreer | first1 = J. G. | year = 1957 | title = A question of terminology | journal = IRE Transactions on Information Theory| volume = 3 | issue = 3 | page = 208 | doi = 10.1109/TIT.1957.1057418}}</ref> Mutual Information is also known as [[information gain]].

== Definition ==
Let <math>(X,Y)</math> be a pair of [[Random variable|random variables]] with values over the space <math>\mathcal{X}\times\mathcal{Y}</math>. If their joint distribution is <math>P_{(X,Y)}</math> and the marginal distributions are <math>P_X</math> and <math>P_Y</math>, the mutual information is defined as

:<math>I(X;Y) = D_{\mathrm{KL}}( P_{(X,Y)} \| P_{X} \otimes P_{Y} )</math>

where <math>D_{\mathrm{KL}}</math> is the [[Kullback–Leibler divergence]], and <math>P_{X} \otimes P_{Y}</math> is the [[outer product]] distribution which assigns probability <math>P_X(x)\cdot P_Y(y)</math> to each <math>(x,y)</math>.

Expressed in terms of the [[Entropy (information theory)|entropy]] <math>H(\cdot)</math> and the [[conditional entropy]] <math>H(\cdot|\cdot)</math> of the random variables <math>X</math> and <math>Y</math>, one also has (See [[Mutual information#Relation to conditional and joint entropy|relation to conditional and joint entropy]]):
:<math>I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)</math>

Notice, as per property of the [[Kullback–Leibler divergence]], that <math>I(X;Y)</math> is equal to zero precisely when the joint distribution coincides with the product of the marginals, i.e. when <math>X</math> and <math>Y</math> are independent (and hence observing <math>Y</math> tells you nothing about <math>X</math>). <math>I(X;Y)</math> is non-negative, it is a measure of the price for encoding <math>(X,Y)</math> as a pair of independent random variables when in reality they are not.

If the [[natural logarithm]] is used, the unit of mutual information is the [[nat (unit)|nat]].  If the [[logarithm|log base]] 2 is used, the unit of mutual information is the [[Shannon (unit)|shannon]], also known as the bit.  If the [[logarithm|log base]] 10 is used, the unit of mutual information is the [[Hartley (unit)|hartley]], also known as the ban or the dit.

=== In terms of PMFs for discrete distributions ===
The mutual information of two jointly discrete random variables <math>X</math> and <math>Y</math> is calculated as a double sum:<ref name=cover1991>{{cite book|last1=Cover|first1=T.M.|last2=Thomas|first2=J.A.|title=Elements of Information Theory|url=https://archive.org/details/elementsofinform0000cove|url-access=registration|date=1991|publisher=John Wiley & Sons |isbn=978-0-471-24195-9|edition=Wiley}}</ref>{{rp|20}}

:<math>
  \operatorname{I}(X; Y) = \sum_{y \in \mathcal Y} \sum_{x \in \mathcal X}
    { P_{(X,Y)}(x, y) \log\left(\frac{P_{(X,Y)}(x, y)}{P_X(x)\,P_Y(y)}\right) }
</math>,

where <math>P_{(X,Y)}</math> is the [[joint distribution|joint probability ''mass'' function]] of <math>X</math> and <math>Y</math>, and <math>P_X</math> and <math>P_Y</math> are the [[marginal probability]] mass functions of <math>X</math> and <math>Y</math> respectively.

=== In terms of PDFs for continuous distributions ===
In the case of jointly continuous random variables, the double sum is replaced by a [[double integral]]:<ref name=cover1991 />{{rp|251}}

:<math>
  \operatorname{I}(X;Y) = 
  \int_{\mathcal Y} \int_{\mathcal X}
      {P_{(X,Y)}(x,y) \log{ \left(\frac{P_{(X,Y)}(x,y)}{P_X(x)\,P_Y(y)} \right) }
  } \; dx \,dy
</math>,

where <math>P_{(X,Y)}</math> is now the joint probability ''density'' function of <math>X</math> and <math>Y</math>, and <math>P_X</math> and <math>P_Y</math> are the marginal probability density functions of <math>X</math> and <math>Y</math> respectively.

== Motivation ==
Intuitively, mutual information measures the information that <math>X</math> and <math>Y</math> share: It measures how much knowing one of these variables reduces uncertainty about the other. For example, if <math>X</math> and <math>Y</math> are independent, then knowing <math>X</math> does not give any information about <math>Y</math> and vice versa, so their mutual information is zero.  At the other extreme, if <math>X</math> is a deterministic function of <math>Y</math> and <math>Y</math> is a deterministic function of <math>X</math> then all information conveyed by <math>X</math> is shared with <math>Y</math>: knowing <math>X</math> determines the value of <math>Y</math> and vice versa. As a result, the mutual information is the same as the uncertainty contained in <math>Y</math> (or <math>X</math>) alone, namely the [[information entropy|entropy]] of <math>Y</math> (or <math>X</math>). A very special case of this is when <math>X</math> and <math>Y</math> are the same random variable.

Mutual information is a measure of the inherent dependence expressed in the [[joint distribution]] of <math>X</math> and <math>Y</math> relative to the marginal distribution of <math>X</math> and <math>Y</math> under the assumption of independence. Mutual information therefore measures dependence in the following sense: <math>\operatorname{I}(X;Y) = 0</math> [[if and only if]] <math>X</math> and <math>Y</math> are independent random variables.  This is easy to see in one direction: if <math>X</math> and <math>Y</math> are independent, then <math>p_{(X,Y)}(x,y)=p_X(x) \cdot p_Y(y)</math>, and therefore:

:<math> \log{ \left( \frac{p_{(X,Y)}(x,y)}{p_X(x)\,p_Y(y)} \right) } = \log 1 = 0</math>.

Moreover, mutual information is nonnegative (i.e. <math>\operatorname{I}(X;Y) \ge 0</math> see below) and [[symmetric function|symmetric]] (i.e. <math>\operatorname{I}(X;Y) = \operatorname{I}(Y;X)</math> see below).

== Properties ==
=== Nonnegativity ===
Using [[Jensen's inequality]] on the definition of mutual information we can show that <math>\operatorname{I}(X;Y)</math> is non-negative, i.e.<ref name=cover1991 />{{rp|28}}
:<math>\operatorname{I}(X;Y) \ge 0</math>

=== Symmetry===
:<math>\operatorname{I}(X;Y) = \operatorname{I}(Y;X)</math>
The proof is given considering the relationship with entropy, as shown below.

=== Supermodularity under independence===
If <math> C </math> is independent of <math> (A,B) </math>, then
:<math>\operatorname{I}(Y;A,B,C) - \operatorname{I}(Y;A,B) \ge \operatorname{I}(Y;A,C) - \operatorname{I}(Y;A) </math>.<ref>{{cite journal |last1=Janssen |first1=Joseph |last2=Guan |first2=Vincent |last3=Robeva |first3=Elina |title=Ultra-marginal Feature Importance: Learning from Data with Causal Guarantees |journal=International Conference on Artificial Intelligence and Statistics |date=2023 |pages=10782–10814 |arxiv=2204.09938 |url=https://proceedings.mlr.press/v206/janssen23a.html}}</ref>

=== Relation to conditional and joint entropy ===
Mutual information can be equivalently expressed as:

:<math>\begin{align}
  \operatorname{I}(X;Y) &{} \equiv \Eta(X) - \Eta(X\mid Y) \\
         &{} \equiv \Eta(Y) - \Eta(Y\mid X) \\
         &{} \equiv \Eta(X) + \Eta(Y) - \Eta(X, Y) \\
         &{} \equiv \Eta(X, Y) - \Eta(X\mid Y) - \Eta(Y\mid X)
\end{align}</math>

where <math>\Eta(X)</math> and <math>\Eta(Y)</math> are the marginal [[information entropy|entropies]], <math>\Eta(X\mid Y)</math> and <math>\Eta(Y\mid X)</math> are the [[conditional entropy|conditional entropies]], and <math>\Eta(X,Y)</math> is the [[joint entropy]] of <math>X</math> and <math>Y</math>.

Notice the analogy to the union, difference, and intersection of two sets: in this respect, all the formulas given above are apparent from the Venn diagram reported at the beginning of the article.

In terms of a communication channel in which the output <math>Y</math> is a noisy version of the input <math>X</math>, these relations are summarised in the figure:

[[File:Figchannel2017ab.svg|thumb|The relationships between information theoretic quantities]]

Because <math>\operatorname{I}(X;Y)</math> is non-negative, consequently, <math>\Eta(X) \ge \Eta(X\mid Y)</math>. Here we give the detailed deduction of <math>\operatorname{I}(X;Y)=\Eta(Y)-\Eta(Y\mid X)</math> for the case of jointly discrete random variables:

:<math>
\begin{align}
\operatorname{I}(X;Y) & {} = \sum_{x \in \mathcal{X},y \in \mathcal{Y}} p_{(X,Y)}(x,y) \log \frac{p_{(X,Y)}(x,y)}{p_X(x)p_Y(y)}\\
& {} = \sum_{x \in \mathcal{X},y \in \mathcal{Y}} p_{(X,Y)}(x,y) \log \frac{p_{(X,Y)}(x,y)}{p_X(x)} - \sum_{x \in \mathcal{X},y \in \mathcal{Y}} p_{(X,Y)}(x,y) \log p_Y(y)  \\

& {} = \sum_{x \in \mathcal{X},y \in \mathcal{Y}} p_X(x)p_{Y\mid X=x}(y) \log p_{Y\mid X=x}(y) - \sum_{x \in \mathcal{X},y \in \mathcal{Y}} p_{(X,Y)}(x,y) \log p_Y(y) \\
& {} = \sum_{x \in \mathcal{X}} p_X(x) \left(\sum_{y \in \mathcal{Y}} p_{Y\mid X=x}(y) \log p_{Y\mid X=x}(y)\right) - \sum_{y \in \mathcal{Y}} \left(\sum_{x \in \mathcal{X}} p_{(X,Y)}(x,y)\right) \log p_Y(y) \\
& {} = -\sum_{x \in \mathcal{X}} p_X(x) \Eta(Y\mid X=x) - \sum_{y \in \mathcal{Y}} p_Y(y) \log p_Y(y) \\
& {} = -\Eta(Y\mid X) + \Eta(Y)  \\
& {} = \Eta(Y) - \Eta(Y\mid X).  \\
\end{align}
</math>

The proofs of the other identities above are similar. The proof of the general case (not just discrete) is similar, with integrals replacing sums.

Intuitively, if entropy <math>\Eta(Y)</math> is regarded as a measure of uncertainty about a random variable, then <math>\Eta(Y\mid X)</math> is a measure of what <math>X</math> does ''not'' say about <math>Y</math>. This is "the amount of uncertainty remaining about <math>Y</math> after <math>X</math> is known", and thus the right side of the second of these equalities can be read as "the amount of uncertainty in <math>Y</math>, minus the amount of uncertainty in <math>Y</math> which remains after <math>X</math> is known", which is equivalent to "the amount of uncertainty in <math>Y</math> which is removed by knowing <math>X</math>". This corroborates the intuitive meaning of mutual information as the amount of information (that is, reduction in uncertainty) that knowing either variable provides about the other.

Note that in the discrete case <math>\Eta(Y\mid Y) = 0</math> and therefore <math>\Eta(Y) = \operatorname{I}(Y;Y)</math>. Thus <math>\operatorname{I}(Y; Y) \ge \operatorname{I}(X; Y)</math>, and one can formulate the basic principle that a variable contains at least as much information about itself as any other variable can provide.

=== Relation to Kullback–Leibler divergence ===
For jointly discrete or jointly continuous pairs <math>(X,Y)</math>, mutual information is the [[Kullback–Leibler divergence]] from the product of the [[marginal distribution]]s, <math>p_X \cdot p_Y</math>, of the [[joint distribution]] <math>p_{(X,Y)}</math>, that is,

:<math>\operatorname{I}(X; Y) = D_\text{KL}\left(p_{(X,Y)} \parallel p_Xp_Y\right)</math>

Furthermore, let <math> p_{(X,Y)}(x,y) =p_{X\mid Y=y}(x)* p_Y(y)</math> be the conditional mass or density function. Then, we have the identity

:<math>\operatorname{I}(X; Y) = \mathbb{E}_Y\left[D_\text{KL}\!\left(p_{X\mid Y} \parallel p_X\right)\right]</math>

The proof for jointly discrete random variables is as follows:
:<math>
\begin{align}
  \operatorname{I}(X; Y) &= \sum_{y \in \mathcal Y} \sum_{x \in \mathcal X}
    { p_{(X,Y)}(x, y) \log\left(\frac{p_{(X,Y)}(x, y)}{p_X(x)\,p_Y(y)}\right) } \\
    &= \sum_{y \in \mathcal{Y}} \sum_{x \in \mathcal{X}} p_{X\mid Y=y}(x)  p_Y(y) \log \frac{p_{X\mid Y=y}(x) p_Y(y)}{p_X(x)  p_Y(y)} \\
    &= \sum_{y \in \mathcal{Y}} p_Y(y) \sum_{x \in \mathcal{X}} p_{X\mid Y=y}(x) \log \frac{p_{X\mid Y=y}(x)}{p_X(x)} \\
    &= \sum_{y \in \mathcal{Y}} p_Y(y) \; D_\text{KL}\!\left(p_{X\mid Y=y} \parallel p_X\right) \\
    &= \mathbb{E}_Y \left[D_\text{KL}\!\left(p_{X\mid Y} \parallel p_X\right)\right].
\end{align}
</math>
Similarly this identity can be established for jointly continuous random variables.

Note that here the Kullback–Leibler divergence involves integration over the values of the random variable <math>X</math> only, and the expression <math>D_\text{KL}(p_{X\mid Y} \parallel p_X)</math> still denotes a random variable because <math>Y</math> is random. Thus mutual information can also be understood as the [[expected value|expectation]] of the Kullback–Leibler divergence of the [[univariate distribution]] <math>p_X</math> of <math>X</math> from the [[conditional distribution]] <math>p_{X\mid Y}</math> of <math>X</math> given <math>Y</math>: the more different the distributions <math>p_{X\mid Y}</math> and <math>p_X</math> are on average, the greater the [[Kullback–Leibler divergence|information gain]].

=== Bayesian estimation of mutual information ===

If samples from a joint distribution are available, a Bayesian approach can be used to estimate the mutual information of that distribution. The first work to do this, which also showed how to do Bayesian estimation of many other information-theoretic properties besides mutual information, was.<ref>{{cite journal | last1 = Wolpert | first1 = D.H. | last2 = Wolf | first2 = D.R. | year = 1995 | title = Estimating functions of probability distributions from a finite set of samples | journal = Physical Review E | volume = 52 | issue = 6 | pages = 6841–6854 | doi = 10.1103/PhysRevE.52.6841 | pmid = 9964199 | citeseerx = 10.1.1.55.7122 | bibcode = 1995PhRvE..52.6841W | s2cid = 9795679 }}</ref> Subsequent researchers have rederived <ref>{{cite journal | last1 = Hutter | first1 = M. | year = 2001 | title = Distribution of Mutual Information | journal = Advances in Neural Information Processing Systems }}</ref> and extended <ref>{{cite journal | last1 = Archer | first1 = E. | last2 = Park | first2 = I.M. | last3 = Pillow | first3 = J. | year = 2013 | title = Bayesian and Quasi-Bayesian Estimators for Mutual Information from Discrete Data | journal = Entropy| volume = 15 | issue = 12 | pages = 1738–1755 | doi = 10.3390/e15051738 | citeseerx = 10.1.1.294.4690 | bibcode = 2013Entrp..15.1738A | doi-access = free }}</ref>
this analysis. See <ref>{{cite journal | last1 = Wolpert | first1 = D.H | last2 = DeDeo | first2 = S. | year = 2013 | title = Estimating Functions of Distributions Defined over Spaces of Unknown Size | journal = Entropy | volume = 15 | issue = 12 | pages = 4668–4699 | doi = 10.3390/e15114668 | arxiv = 1311.4548 | bibcode = 2013Entrp..15.4668W | s2cid = 2737117 | doi-access = free }}</ref> for a recent paper based on a prior specifically tailored to estimation of mutual
information per se. Besides, recently an estimation method accounting for continuous and multivariate outputs,  <math>Y</math>, was proposed in 
.<ref>{{citation| journal = [[PLOS Computational Biology]]|volume = 15|issue = 7|pages = e1007132|doi = 10.1371/journal.pcbi.1007132|pmid = 31299056|pmc = 6655862|title=Information-theoretic analysis of multivariate single-cell signaling responses|author1= Tomasz Jetka|author2= Karol Nienaltowski|author3= Tomasz Winarski| author4=Slawomir Blonski| author5= Michal Komorowski|year=2019|bibcode = 2019PLSCB..15E7132J|arxiv = 1808.05581 | doi-access=free }}</ref>

=== Independence assumptions ===
The Kullback-Leibler divergence formulation of the mutual information is predicated on that one is interested in comparing <math>p(x,y)</math> to the fully factorized [[outer product]] <math>p(x) \cdot p(y)</math>. In many problems, such as [[non-negative matrix factorization]], one is interested in less extreme factorizations; specifically, one wishes to compare <math>p(x,y)</math> to a low-rank matrix approximation in some unknown variable <math>w</math>; that is, to what degree one might have
: <math>p(x,y)\approx \sum_w p^\prime (x,w) p^{\prime\prime}(w,y)</math>

Alternately, one might be interested in knowing how much more information <math>p(x,y)</math> carries over its factorization. In such a case, the excess information that the full distribution <math>p(x,y)</math> carries over the matrix factorization is given by the Kullback-Leibler divergence
:<math>\operatorname{I}_{LRMA} = \sum_{y \in \mathcal{Y}} \sum_{x \in \mathcal{X}}
    {p(x,y) \log{ \left(\frac{p(x,y)}{\sum_w p^\prime (x,w) p^{\prime\prime}(w,y)}
      \right) }},
</math>

The conventional definition of the mutual information is recovered in the extreme case that the process <math>W</math> has only one value for <math>w</math>.

== Variations ==
Several variations on mutual information have been proposed to suit various needs.  Among these are normalized variants and generalizations to more than two variables.

=== Metric ===
Many applications require a [[metric (mathematics)|metric]], that is, a distance measure between pairs of points. The quantity

:<math>\begin{align}
  d(X,Y) &= \Eta(X,Y) - \operatorname{I}(X;Y) \\
         &= \Eta(X) + \Eta(Y) - 2\operatorname{I}(X;Y) \\
         &= \Eta(X\mid Y) + \Eta(Y\mid X) \\
         &= 2\Eta(X,Y) - \Eta(X) - \Eta(Y)
\end{align}</math>

satisfies the properties of a metric ([[triangle inequality]], [[non-negative|non-negativity]], [[identity of indiscernibles|indiscernability]] and symmetry), where equality <math>X=Y</math> is understood to mean that <math>X</math> can be completely determined from <math>Y</math>.<ref>{{cite journal | last1 = Rajski | first1 = C. | year = 1961 | title = A metric space of discrete probability distributions | journal = Information and Control| volume = 4 | issue = 4 | pages = 371–377 | doi = 10.1016/S0019-9958(61)80055-7| doi-access =  }}</ref>

This distance metric is also known as the [[variation of information]].

If <math>X, Y</math> are discrete random variables then all the entropy terms are non-negative, so <math>0 \le d(X,Y) \le \Eta(X,Y)</math> and one can define a normalized distance

:<math>D(X,Y) = \frac{d(X, Y)}{\Eta(X, Y)} \le 1.</math>

Plugging in the definitions shows that

:<math>D(X,Y) = 1 - \frac{\operatorname{I}(X; Y)}{\Eta(X, Y)}.</math>

This is known as the Rajski Distance.<ref>{{cite journal | last1 = Rajski | first1 = C. | year = 1961 | title = A metric space of discrete probability distributions | journal = Information and Control| volume = 4 | issue = 4 | pages = 371–377 | doi = 10.1016/S0019-9958(61)80055-7| doi-access =  }}</ref> In a set-theoretic interpretation of information (see the figure for [[Conditional entropy]]), this is effectively the [[Jaccard index|Jaccard distance]] between <math>X</math> and <math>Y</math>.

Finally,

:<math>D^\prime(X, Y) = 1 - \frac{\operatorname{I}(X; Y)}{\max\left\{\Eta(X), \Eta(Y)\right\}}</math>

is also a metric.

=== Conditional mutual information ===
{{Main|Conditional mutual information}}
Sometimes it is useful to express the mutual information of two random variables conditioned on a third.

:<math>\operatorname{I}(X;Y|Z) = \mathbb{E}_Z [D_{\mathrm{KL}}( P_{(X,Y)|Z} \| P_{X|Z} \otimes P_{Y|Z} )]</math>

For jointly [[discrete random variable]]s this takes the form
:<math>
\operatorname{I}(X;Y|Z)  =  \sum_{z\in \mathcal{Z}} \sum_{y\in \mathcal{Y}} \sum_{x\in \mathcal{X}}
    {p_Z(z)\, p_{X,Y|Z}(x,y|z) 
        \log\left[\frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}\,(x|z)p_{Y|Z}(y|z)}\right]},
</math>

which can be simplified as
:<math>
\operatorname{I}(X;Y|Z) = \sum_{z\in \mathcal{Z}} \sum_{y\in \mathcal{Y}} \sum_{x\in \mathcal{X}}
      p_{X,Y,Z}(x,y,z) \log \frac{p_{X,Y,Z}(x,y,z)p_{Z}(z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)}.
</math>

For jointly [[continuous random variable]]s this takes the form
:<math>
\operatorname{I}(X;Y|Z)  =  \int_{\mathcal{Z}} \int_{\mathcal{Y}} \int_{\mathcal{X}}
    {p_Z(z)\, p_{X,Y|Z}(x,y|z) 
        \log\left[\frac{p_{X,Y|Z}(x,y|z)}{p_{X|Z}\,(x|z)p_{Y|Z}(y|z)}\right]} dx dy dz,
</math>

which can be simplified as
:<math>
\operatorname{I}(X;Y|Z) = \int_{\mathcal{Z}} \int_{\mathcal{Y}} \int_{\mathcal{X}}
      p_{X,Y,Z}(x,y,z) \log \frac{p_{X,Y,Z}(x,y,z)p_{Z}(z)}{p_{X,Z}(x,z)p_{Y,Z}(y,z)} dx dy dz.
</math>

Conditioning on a third random variable may either increase or decrease the mutual information, but it is always true that
:<math>\operatorname{I}(X;Y|Z) \ge 0</math>

for discrete, jointly distributed random variables <math>X,Y,Z</math>. This result has been used as a basic building block for proving other [[inequalities in information theory]].

=== Interaction information ===
{{Main|Interaction information}}
Several generalizations of mutual information to more than two random variables have been proposed, such as [[total correlation]] (or multi-information) and [[dual total correlation]]. The expression and study of multivariate higher-degree mutual information was achieved in two seemingly independent works: McGill (1954)<ref>{{cite journal | last1 = McGill| first1 = W. | year = 1954 | title = Multivariate information transmission | journal = Psychometrika | volume = 19 | issue = 1 | pages = 97–116 | doi = 10.1007/BF02289159 | s2cid = 126431489 }}</ref> who called these functions "[[interaction information]]", and Hu Kuo Ting (1962).<ref name="On the Amount of Information">{{cite journal | last1 = Hu| first1 = K.T. | year = 1962 | title = On the Amount of Information | journal = Theory Probab. Appl. | volume = 7 | issue = 4 |  pages = 439–447 | doi = 10.1137/1107041 }}</ref> Interaction information is defined for one variable as follows:
:<math>\operatorname{I}(X_1) = \Eta(X_1)</math>

and for <math>n > 1,</math>
:<math>
    \operatorname{I}(X_1;\,...\,;X_n)
  = \operatorname{I}(X_1;\,...\,;X_{n-1}) 
      - \operatorname{I}(X_1;\,...\,;X_{n-1}\mid X_n).
</math>

Some authors reverse the order of the terms on the right-hand side of the preceding equation, which changes the sign when the number of random variables is odd. (And in this case, the single-variable expression becomes the negative of the entropy.) Note that
:<math>
  I(X_1;\ldots;X_{n-1}\mid X_{n}) =
  \mathbb{E}_{X_{n}} [D_{\mathrm{KL}}( P_{(X_1,\ldots,X_{n-1})\mid X_{n}} \| P_{X_1\mid X_{n}} \otimes\cdots\otimes P_{X_{n-1}\mid X_{n}} )].
</math>

====Multivariate statistical independence ====
The multivariate mutual information functions generalize the pairwise independence case that states that <math>X_1, X_2</math> if and only if <math>I(X_1; X_2) = 0</math>, to arbitrary numerous variable. n variables are mutually independent if and only if the <math>2^n - n - 1</math> mutual information functions vanish <math>I(X_1; \ldots; X_k) = 0</math> with <math>n \ge k \ge 2</math> (theorem 2<ref name="e21090869">{{cite journal|last1=Baudot|first1=P.|last2=Tapia|first2=M.|last3=Bennequin|first3=D.|last4=Goaillard|first4=J.M.|year=2019|title=Topological Information Data Analysis|journal=Entropy|volume=21|issue=9|at=869|arxiv=1907.04242|bibcode=2019Entrp..21..869B|doi=10.3390/e21090869| pmc=7515398|s2cid=195848308|doi-access=free}}</ref>). In this sense, the <math>I(X_1; \ldots; X_k) = 0</math> can be used as a refined statistical independence criterion.

==== Applications ====
For 3 variables, Brenner et al. applied multivariate mutual information to [[neural coding]] and called its negativity "synergy"<ref>{{cite journal | last1 = Brenner | first1 = N. | last2 = Strong | first2 = S. | last3 = Koberle | first3 = R. | last4 = Bialek | first4 = W. | year = 2000 | title = Synergy in a Neural Code | doi = 10.1162/089976600300015259 | pmid = 10935917 | journal = Neural Comput | volume = 12 | issue = 7 | pages = 1531–1552 | s2cid = 600528 }}</ref> and Watkinson et al. applied it to genetic expression.<ref>{{cite journal | last1 = Watkinson | first1 = J. | last2 = Liang | first2 = K. | last3 = Wang | first3 = X. | last4 = Zheng | first4 = T.| last5 = Anastassiou | first5 = D. | year = 2009 | title = Inference of Regulatory Gene Interactions from Expression Data Using Three-Way Mutual Information | doi = 10.1111/j.1749-6632.2008.03757.x | pmid = 19348651 | journal = Chall. Syst. Biol. Ann. N. Y. Acad. Sci. | volume = 1158 | issue = 1 | pages = 302–313 | bibcode = 2009NYASA1158..302W | s2cid = 8846229 }}</ref> For arbitrary k variables, Tapia et al. applied multivariate mutual information to gene expression.<ref name=s41598>{{cite journal|last1=Tapia|first1=M.|last2=Baudot|first2=P.|last3=Formizano-Treziny|first3=C.|last4=Dufour|first4=M.|last5=Goaillard|first5=J.M.|year=2018|title=Neurotransmitter identity and electrophysiological phenotype are genetically coupled in midbrain dopaminergic neurons|doi= 10.1038/s41598-018-31765-z|pmid=30206240|pmc=6134142|journal=Sci. Rep.|volume=8|issue=1|pages=13637|bibcode=2018NatSR...813637T}}</ref><ref name=e21090869/> It can be zero, positive, or negative.<ref name="On the Amount of Information"/> The positivity corresponds to relations generalizing the pairwise correlations, nullity corresponds to a refined notion of independence, and negativity detects high dimensional "emergent" relations and clusterized datapoints <ref name=s41598/>).

One high-dimensional generalization scheme which maximizes the mutual information between the joint distribution and other target variables is found to be useful in [[feature selection]].<ref>{{cite book
  | author1=Christopher D. Manning | author2=Prabhakar Raghavan | author3=Hinrich Schütze
  | title = An Introduction to Information Retrieval
  | publisher = [[Cambridge University Press]]
  | year = 2008
  | isbn = 978-0-521-86571-5
}}</ref>

Mutual information is also used in the area of signal processing as a [[similarity measure|measure of similarity]] between two signals. For example, FMI metric<ref>{{cite journal | last1 = Haghighat | first1 = M. B. A. | last2 = Aghagolzadeh | first2 = A. | last3 = Seyedarabi | first3 = H. | year = 2011 | title = A non-reference image fusion metric based on mutual information of image features | doi = 10.1016/j.compeleceng.2011.07.012 | journal = Computers & Electrical Engineering | volume = 37 | issue = 5| pages = 744–756 | s2cid = 7738541 }}</ref> is an image fusion performance measure that makes use of mutual information in order to measure the amount of information that the fused image contains about the source images. The [[Matlab]] code for this metric can be found at.<ref>{{cite web|url=http://www.mathworks.com/matlabcentral/fileexchange/45926-feature-mutual-information-fmi-image-fusion-metric|title=Feature Mutual Information (FMI) metric for non-reference image fusion - File Exchange - MATLAB Central|website=www.mathworks.com|access-date=4 April 2018}}</ref> A python package for computing all multivariate mutual informations, [[conditional mutual information]], joint entropies, total correlations, information distance in a dataset of n variables is available.<ref>{{cite web|url=https://infotopo.readthedocs.io/en/latest/index.html|title=InfoTopo: Topological Information Data Analysis. Deep statistical unsupervised and supervised learning - File Exchange - Github|website=github.com/pierrebaudot/infotopopy/|access-date=26 September 2020}}</ref>

=== Directed information ===
[[Directed information]], <math>\operatorname{I}\left(X^n \to Y^n\right)</math>, measures the amount of information that flows from the process <math>X^n</math> to <math>Y^n</math>, where <math>X^n</math> denotes the vector <math>X_1, X_2, ..., X_n</math> and <math>Y^n</math> denotes <math>Y_1, Y_2, ..., Y_n</math>. The term ''directed information'' was coined by [[James Massey]] and is defined as
:<math>
    \operatorname{I}\left(X^n \to Y^n\right)
  = \sum_{i=1}^n \operatorname{I}\left(X^i; Y_i\mid Y^{i-1}\right)
</math>.

Note that if <math>n=1</math>, the directed information becomes the mutual information. Directed information has many applications in problems where [[causality]] plays an important role, such as [[Channel capacity|capacity of channel]] with feedback.<ref>{{cite conference|last1=Massey|first1=James|title=Causality, Feedback And Directed Informatio|date=1990|book-title=Proc. 1990 Intl. Symp. on Info. Th. and its Applications, Waikiki, Hawaii, Nov. 27-30, 1990|citeseerx=10.1.1.36.5688}}</ref><ref>{{cite journal|last1=Permuter|first1=Haim Henry|last2=Weissman|first2=Tsachy|last3=Goldsmith|first3=Andrea J.|title=Finite State Channels With Time-Invariant Deterministic Feedback|journal=IEEE Transactions on Information Theory|date=February 2009|volume=55|issue=2|pages=644–662|doi=10.1109/TIT.2008.2009849|arxiv=cs/0608070|s2cid=13178}}</ref>

=== Normalized variants ===
Normalized variants of the mutual information are provided by the ''coefficients of constraint'',{{sfn|Coombs|Dawes|Tversky|1970}} [[uncertainty coefficient]]<ref name=pressflannery>{{Cite book|last1=Press|first1=WH|last2=Teukolsky|first2=SA|last3=Vetterling|first3=WT|last4=Flannery|first4=BP|year=2007|title=Numerical Recipes: The Art of Scientific Computing|edition=3rd|publisher=Cambridge University Press|location=New York|isbn=978-0-521-88068-8|chapter=Section 14.7.3. Conditional Entropy and Mutual Information|chapter-url=http://apps.nrbook.com/empanel/index.html#pg=758|access-date=2011-08-13|archive-date=2011-08-11|archive-url=https://web.archive.org/web/20110811154417/http://apps.nrbook.com/empanel/index.html#pg=758|url-status=dead}}</ref> or proficiency:<ref name=JimWhite>{{Cite conference | last1= White | first1= Jim | last2= Steingold | first2= Sam | last3= Fournelle | first3= Connie | title= Performance Metrics for Group-Detection Algorithms | conference= Interface 2004 | url= http://www.interfacesymposia.org/I04/I2004Proceedings/WhiteJim/WhiteJim.paper.pdf | access-date= 2014-02-19 | archive-date= 2016-07-05 | archive-url= https://web.archive.org/web/20160705074827/http://www.interfacesymposia.org/I04/I2004Proceedings/WhiteJim/WhiteJim.paper.pdf | url-status= usurped }}</ref>
:<math>
  C_{XY} = \frac{\operatorname{I}(X;Y)}{\Eta(Y)}
  ~~~~\mbox{and}~~~~ 
  C_{YX} = \frac{\operatorname{I}(X;Y)}{\Eta(X)}.
</math>

The two coefficients have a value ranging in [0, 1], but are not necessarily equal. This measure is not symmetric. If one desires a symmetric measure they can consider the following ''[[Redundancy (information theory)|redundancy]]'' measure:
:<math>R = \frac{\operatorname{I}(X;Y)}{\Eta(X) + \Eta(Y)}</math>

which attains a minimum of zero when the variables are independent and a maximum value of
:<math>R_\max = \frac{\min\left\{\Eta(X), \Eta(Y)\right\}}{\Eta(X) + \Eta(Y)}</math>

when one variable becomes completely redundant with the knowledge of the other.  See also ''[[Redundancy (information theory)]]''.

Another symmetrical measure is the ''symmetric uncertainty'' {{harv|Witten|Frank|2005}}, given by
:<math>U(X, Y) = 2R = 2\frac{\operatorname{I}(X;Y)}{\Eta(X) + \Eta(Y)}</math>

which represents the [[harmonic mean]] of the two uncertainty coefficients <math>C_{XY}, C_{YX}</math>.<ref name=pressflannery />

If we consider mutual information as a special case of the [[total correlation]] or [[dual total correlation]], the normalized version are respectively,
:<math>\frac{\operatorname{I}(X; Y)}{\min\left[\Eta(X), \Eta(Y)\right]}</math> and <math>\frac{\operatorname{I}(X; Y)}{\Eta(X, Y)}\; .</math>

This normalized version also known as '''Information Quality Ratio (IQR)''' which quantifies the amount of information of a variable based on another variable against total uncertainty:<ref name=DRWijaya>{{Cite journal
 | last1= Wijaya | first1= Dedy Rahman
 | last2= Sarno | first2= Riyanarto
 | last3= Zulaika | first3= Enny
 | title = Information Quality Ratio as a novel metric for mother wavelet selection
 | journal = Chemometrics and Intelligent Laboratory Systems
 | volume = 160
 | pages = 59–71
 | doi = 10.1016/j.chemolab.2016.11.012
 | year= 2017
}}</ref>
:<math>
    IQR(X, Y) = \operatorname{E}[\operatorname{I}(X;Y)] 
  = \frac{\operatorname{I}(X;Y)}{\Eta(X, Y)} 
  = \frac{\sum_{x \in X} \sum_{y \in Y} p(x, y) \log {p(x)p(y)}}{\sum_{x \in X} \sum_{y \in Y} p(x, y) \log {p(x, y)}} - 1
</math>

There's a normalization<ref name="strehl-jmlr02">{{cite journal
 | title = Cluster Ensembles – A Knowledge Reuse Framework for Combining Multiple Partitions
 | journal = The Journal of Machine Learning Research
 | pages = 583–617 | volume = 3 | year = 2003
 | last1 = Strehl | first1 = Alexander
 | last2 = Ghosh | first2 = Joydeep
 | doi=10.1162/153244303321897735
 | url=http://www.jmlr.org/papers/volume3/strehl02a/strehl02a.pdf
}}</ref> which derives from first thinking of mutual information as an analogue to [[covariance]] (thus [[Entropy (information theory)|Shannon entropy]] is analogous to [[variance]]).  Then the normalized mutual information is calculated akin to the [[Pearson product-moment correlation coefficient|Pearson correlation coefficient]],

:<math>
\frac{\operatorname{I}(X;Y)}{\sqrt{\Eta(X)\Eta(Y)}}\; .
</math>

=== Weighted variants ===
In the traditional formulation of the mutual information,

:<math>
    \operatorname{I}(X;Y) 
  = \sum_{y \in Y} \sum_{x \in X} p(x, y) \log \frac{p(x, y)}{p(x)\,p(y)},
</math>

each ''event'' or ''object''  specified by <math>(x, y)</math> is weighted by the corresponding  probability <math>p(x, y)</math>.  This assumes that all objects or events are equivalent ''apart from'' their probability of occurrence.  However, in some applications it may be the case that certain objects or events are more ''significant'' than others, or that certain patterns of association are more semantically important than others.

For example, the deterministic mapping <math>\{(1,1),(2,2),(3,3)\}</math> may be viewed as stronger than the deterministic mapping <math>\{(1,3),(2,1),(3,2)\}</math>, although these relationships would yield the same mutual information.  This is because the mutual information is not sensitive at all to any inherent ordering in the variable values ({{harvnb|Cronbach|1954}}, {{harvnb|Coombs|Dawes|Tversky|1970}}, {{harvnb|Lockhead|1970}}), and is therefore not sensitive at all to the '''form''' of the relational mapping between the associated variables.  If it is desired that the former relation—showing agreement on all variable values—be judged stronger than the later relation, then it is possible to use the following ''weighted mutual information'' {{harv|Guiasu|1977}}.
:<math>
    \operatorname{I}(X;Y) 
  = \sum_{y \in Y} \sum_{x \in X} w(x,y) p(x,y) \log \frac{p(x,y)}{p(x)\,p(y)},
</math>

which places a weight <math>w(x,y)</math> on the probability of each variable value co-occurrence, <math>p(x,y)</math>. This allows that certain probabilities may carry more or less significance than others, thereby allowing the quantification of relevant ''holistic'' or ''[[Prägnanz]]'' factors.  In the above example, using larger relative weights for <math>w(1,1)</math>, <math>w(2,2)</math>, and <math>w(3,3)</math> would have the effect of assessing greater ''informativeness'' for the relation  <math>\{(1,1),(2,2),(3,3)\}</math> than for the relation <math>\{(1,3),(2,1),(3,2)\}</math>, which may be desirable in some cases of pattern recognition, and the like.  This weighted mutual information is a form of weighted KL-Divergence, which is known to take negative values for some inputs,<ref name="weighted-kl">{{cite journal | last1 = Kvålseth | first1 = T. O. | year = 1991 | title = The relative useful information measure: some comments | journal = Information Sciences | volume = 56 | issue = 1| pages = 35–38 | doi=10.1016/0020-0255(91)90022-m}}</ref> and there are examples where the weighted mutual information also takes negative values.<ref>{{cite thesis
  |title=Feature Selection Via Joint Likelihood
  |first=A. |last=Pocock
  |year=2012
  |url=http://www.cs.man.ac.uk/~gbrown/publications/pocockPhDthesis.pdf
}}</ref>

=== Adjusted mutual information ===
{{Main|adjusted mutual information}}

A probability distribution can be viewed as a [[partition of a set]].  One may then ask: if a set were partitioned randomly, what would the distribution of probabilities be?  What would the expectation value of the mutual information be? The [[adjusted mutual information]] or AMI subtracts the expectation value of the MI, so that the AMI is zero when two different distributions are random, and one when two distributions are identical.  The AMI is defined in analogy to the [[adjusted Rand index]] of two different partitions of a set.

=== Absolute mutual information ===<!-- This section is linked from [[Kolmogorov complexity]] -->
Using the ideas of [[Kolmogorov complexity]], one can consider the mutual information of two sequences independent of any probability distribution:

:<math>
\operatorname{I}_K(X;Y) = K(X) - K(X\mid Y).
</math>

To establish that this quantity is symmetric up to a logarithmic factor (<math>\operatorname{I}_K(X;Y) \approx \operatorname{I}_K(Y;X)</math>) one requires the [[chain rule for Kolmogorov complexity]] {{Harvard citation|Li|Vitányi|1997}}. Approximations of this quantity via [[Data compression|compression]] can be used to define a [[Metric (mathematics)|distance measure]] to perform a [[hierarchical clustering]] of sequences without having any [[domain knowledge]] of the sequences {{Harvard citation|Cilibrasi|Vitányi|2005}}.

=== Linear correlation ===
Unlike correlation coefficients, such as the [[product moment correlation coefficient]], mutual information contains information about all dependence—linear and nonlinear—and not just linear dependence as the correlation coefficient measures. However, in the narrow case that the joint distribution for <math>X</math> and <math>Y</math> is a [[bivariate normal distribution]] (implying in particular that both marginal distributions are normally distributed), there is an exact relationship between <math>\operatorname{I}</math> and the correlation coefficient <math>\rho</math> {{harv|Gel'fand|Yaglom|1957}}.
:<math>\operatorname{I} = -\frac{1}{2} \log\left(1 - \rho^2\right)</math>

The equation above can be derived as follows for a bivariate Gaussian:
:<math>\begin{align}
  \begin{pmatrix}
    X_1 \\
    X_2
  \end{pmatrix}  &\sim \mathcal{N} \left( \begin{pmatrix}
    \mu_1 \\
    \mu_2
  \end{pmatrix}, \Sigma \right),\qquad
  \Sigma = \begin{pmatrix}
    \sigma^2_1           & \rho\sigma_1\sigma_2 \\
    \rho\sigma_1\sigma_2 & \sigma^2_2
  \end{pmatrix} \\
       \Eta(X_i) &= \frac{1}{2}\log\left(2\pi e \sigma_i^2\right) = \frac{1}{2} + \frac{1}{2}\log(2\pi) + \log\left(\sigma_i\right), \quad i\in\{1, 2\} \\
  \Eta(X_1, X_2) &= \frac{1}{2}\log\left[(2\pi e)^2|\Sigma|\right] = 1 + \log(2\pi) + \log\left(\sigma_1 \sigma_2\right) + \frac{1}{2}\log\left(1 - \rho^2\right) \\
\end{align}</math>

Therefore, 
:<math> 
    \operatorname{I}\left(X_1; X_2\right) 
  = \Eta\left(X_1\right) + \Eta\left(X_2\right) - \Eta\left(X_1, X_2\right) 
  = -\frac{1}{2}\log\left(1 - \rho^2\right)
</math>

=== For discrete data ===
When <math>X</math> and <math>Y</math>  are limited to be in a discrete number of states, observation data is summarized in a [[contingency table]], with row variable <math>X</math> (or <math>i</math>) and column variable <math>Y</math> (or <math>j</math>).  Mutual information is one of the measures of [[association (statistics)|association]] or [[correlation and dependence|correlation]] between the row and column variables.

Other measures of association include [[Pearson's chi-squared test]] statistics, [[G-test]] statistics, etc. In fact, with the same log base, mutual information will be equal to the [[G-test]] log-likelihood statistic divided by <math>2N</math>, where <math>N</math> is the sample size.

== Applications ==
In many applications, one wants to maximize mutual information (thus increasing dependencies), which is often equivalent to minimizing [[conditional entropy]].  Examples include:
* In [[search engine technology]], mutual information between phrases and contexts is used as a feature for [[k-means clustering]] to discover semantic clusters (concepts).<ref name=magerman>[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.4178&rep=rep1&type=pdf Parsing a Natural Language Using Mutual Information Statistics] by David M. Magerman and Mitchell P. Marcus</ref>  For example, the mutual information of a bigram might be calculated as:

:<math>
  MI(x,y) = \log \frac{P_{X,Y}(x,y)}{P_X(x) P_Y(y)} \approx \log \frac{\frac{f_{XY}}{B}}{\frac{f_X}{U} \frac{f_Y}{U}} 
</math>

: where <math>f_{XY}</math> is the number of times the bigram xy appears in the corpus, <math>f_{X}</math> is the number of times the unigram x appears in the corpus, B is the total number of bigrams, and U is the total number of unigrams.<ref name=magerman/>
* In [[telecommunications]], the [[channel capacity]] is equal to the mutual information, maximized over all input distributions.
* [[Discriminative model|Discriminative training]] procedures for [[hidden Markov model]]s have been proposed based on the [[maximum mutual information]] (MMI) criterion.
* [[Nucleic acid secondary structure|RNA secondary structure]] prediction from a [[multiple sequence alignment]].
* [[Phylogenetic profiling]] prediction from pairwise present and disappearance of functionally link [[gene]]s.
* Mutual information has been used as a criterion for [[feature selection]] and feature transformations in [[machine learning]]. It can be used to characterize both the relevance and redundancy of variables, such as the [[minimum redundancy feature selection]].
* Mutual information is used in determining the similarity of two different [[cluster analysis|clusterings]] of a dataset.  As such, it provides some advantages over the traditional [[Rand index]].
* Mutual information of words is often used as a significance function for the computation of [[collocation]]s in [[corpus linguistics]]. This has the added complexity that no word-instance is an instance to two different words; rather, one counts instances where 2 words occur adjacent or in close proximity; this slightly complicates the calculation, since the expected probability of one word occurring within <math>N</math> words of another, goes up with <math>N</math>
* Mutual information is used in [[medical imaging]] for [[image registration]]. Given a reference image (for example, a brain scan), and a second image which needs to be put into the same [[coordinate system]] as the reference image, this image is deformed until the mutual information between it and the reference image is maximized.
* Detection of [[phase synchronization]] in [[time series]] analysis.
* In the [[infomax]] method for neural-net and other machine learning, including the infomax-based [[Independent component analysis]] algorithm
* Average mutual information in [[delay embedding theorem]] is used for determining the ''embedding delay'' parameter.
* Mutual information between [[genes]] in [[microarray|expression microarray]] data is used by the ARACNE algorithm for reconstruction of [[gene regulatory network|gene networks]].
* In [[statistical mechanics]], [[Loschmidt's paradox]] may be expressed in terms of mutual information.<ref name=everett56>[[Hugh Everett]] [https://www.pbs.org/wgbh/nova/manyworlds/pdf/dissertation.pdf Theory of the Universal Wavefunction], Thesis, Princeton University, (1956, 1973), pp 1–140 (page 30)</ref><ref name=everett57>{{cite journal | last1 = Everett | first1 = Hugh | author-link = Hugh Everett | year = 1957 | title = Relative State Formulation of Quantum Mechanics | url = http://www.univer.omsk.su/omsk/Sci/Everett/paper1957.html | journal = Reviews of Modern Physics | volume = 29 | issue = 3 | pages = 454–462 | doi = 10.1103/revmodphys.29.454 | bibcode = 1957RvMP...29..454E | access-date = 2012-07-16 | archive-url = https://web.archive.org/web/20111027191052/http://www.univer.omsk.su/omsk/Sci/Everett/paper1957.html | archive-date = 2011-10-27 | url-status = dead }}</ref> Loschmidt noted that it must be impossible to determine a physical law which lacks [[time reversal symmetry]] (e.g. the [[second law of thermodynamics]]) only from physical laws which have this symmetry. He pointed out that the [[H-theorem]] of [[Boltzmann]] made the assumption that the velocities of particles in a gas  were permanently uncorrelated, which removed the time symmetry inherent in the H-theorem. It can be shown that if a system is described by a probability density in [[phase space]], then [[Liouville's theorem (Hamiltonian)|Liouville's theorem]] implies that the joint information (negative of the joint entropy) of the distribution remains constant in time. The joint information is equal to the mutual information plus the sum of all the marginal information (negative of the marginal entropies) for each particle coordinate. Boltzmann's assumption amounts to ignoring the mutual information in the calculation of entropy, which yields the thermodynamic entropy (divided by the Boltzmann constant).
* In [[Stochastic process|stochastic processes]] coupled to changing environments, mutual information can be used to disentangle internal and effective environmental dependencies.<ref>{{Cite journal |last1=Nicoletti |first1=Giorgio |last2=Busiello |first2=Daniel Maria |date=2021-11-22 |title=Mutual Information Disentangles Interactions from Changing Environments |url=https://link.aps.org/doi/10.1103/PhysRevLett.127.228301 |journal=Physical Review Letters |volume=127 |issue=22 |pages=228301 |doi=10.1103/PhysRevLett.127.228301|pmid=34889638 |arxiv=2107.08985 |bibcode=2021PhRvL.127v8301N |s2cid=236087228 }}</ref><ref>{{Cite journal |last1=Nicoletti |first1=Giorgio |last2=Busiello |first2=Daniel Maria |date=2022-07-29 |title=Mutual information in changing environments: Nonlinear interactions, out-of-equilibrium systems, and continuously varying diffusivities |url=https://link.aps.org/doi/10.1103/PhysRevE.106.014153 |journal=Physical Review E |volume=106 |issue=1 |pages=014153 |doi=10.1103/PhysRevE.106.014153|pmid=35974654 |arxiv=2204.01644 |bibcode=2022PhRvE.106a4153N }}</ref> This is particularly useful when a physical system undergoes changes in the parameters describing its dynamics, e.g., changes in temperature.
* The mutual information is used to learn the structure of [[Bayesian network]]s/[[dynamic Bayesian network]]s, which is thought to explain the causal relationship between random variables, as exemplified by the GlobalMIT toolkit:<ref>{{Google Code|globalmit|GlobalMIT}}</ref> learning the globally optimal dynamic Bayesian network with the Mutual Information Test criterion.
* The mutual information is used to quantify information transmitted during the updating procedure in the [[Gibbs sampling]] algorithm.<ref>{{Cite journal |last=Lee|first=Se Yoon|  title = Gibbs sampler and coordinate ascent variational inference: A set-theoretical review|journal=Communications in Statistics - Theory and Methods|year=2021|volume=51 |issue=6 |pages=1549–1568|doi=10.1080/03610926.2021.1921214|arxiv=2008.01006|s2cid=220935477}}</ref>
* Popular cost function in [[decision tree learning]].
* The mutual information is used in [[cosmology]] to test the influence of large-scale environments on galaxy properties in the [[Galaxy Zoo]].
* The mutual information was used in [[Solar Physics]]  to derive the solar [[differential rotation]] profile, a travel-time deviation map for sunspots, and a time–distance diagram from quiet-Sun measurements<ref>{{cite journal|last1=Keys|first1=Dustin|last2=Kholikov|first2=Shukur|last3=Pevtsov|first3=Alexei A.|title=Application of Mutual Information Methods in Time Distance Helioseismology|journal=Solar Physics|date=February 2015|volume=290|issue=3|pages=659–671|doi=10.1007/s11207-015-0650-y|arxiv=1501.05597|bibcode=2015SoPh..290..659K|s2cid=118472242}}</ref>
* Used in Invariant Information Clustering to automatically train neural network classifiers and image segmenters given no labelled data.<ref name=iic>[https://arxiv.org/abs/1807.06653 Invariant Information Clustering for Unsupervised Image Classification and Segmentation] by Xu Ji, Joao Henriques and Andrea Vedaldi</ref>
* In [[Multiscale modeling|stochastic dynamical systems with multiple timescales]], mutual information has been shown to capture the functional couplings between different temporal scales.<ref>{{Cite journal |last1=Nicoletti |first1=Giorgio |last2=Busiello |first2=Daniel Maria |date=2024-04-08 |title=Information Propagation in Multilayer Systems with Higher-Order Interactions across Timescales |url=https://link.aps.org/doi/10.1103/PhysRevX.14.021007 |journal=Physical Review X |volume=14 |issue=2 |pages=021007 |doi=10.1103/PhysRevX.14.021007|arxiv=2312.06246 |bibcode=2024PhRvX..14b1007N }}</ref> Importantly, it was shown that physical interactions may or may not give rise to mutual information, depending on the typical timescale of their dynamics.

== See also ==
* [[Data differencing]]
* [[Pointwise mutual information]]
* [[Quantum mutual information]]
* [[Specific-information]]

== Notes ==
{{Reflist}}

== References ==
* {{cite journal|last1=Baudot|first1=P.|last2=Tapia|first2=M.|last3=Bennequin|first3=D.|last4=Goaillard|first4=J.M.|title=Topological Information Data Analysis|journal=Entropy|volume=21|issue=9|at=869|year=2019|doi= 10.3390/e21090869| pmc=7515398|bibcode=2019Entrp..21..869B|arxiv=1907.04242|s2cid=195848308|doi-access=free}}
* {{cite journal
  | last1 = Cilibrasi
  | first1 = R.
  | first2 = Paul
  | last2 = Vitányi
  | title = Clustering by compression
  | journal = IEEE Transactions on Information Theory
  | volume = 51
  | issue = 4
  | pages = 1523–1545
  | year = 2005
  | url = http://www.cwi.nl/~paulv/papers/cluster.pdf
  | doi = 10.1109/TIT.2005.844059
  | arxiv = cs/0312044
  | s2cid = 911
  }}
* {{Cite book|last1=Cronbach|first1=L. J.|year=1954|chapter=On the non-rational application of information measures in psychology|editor1-first=Henry|editor1-last=Quastler|editor1-link=Henry Quastler|title=Information Theory in Psychology: Problems and Methods|publisher=Free Press|place=Glencoe, Illinois|pages=14–30}}
* {{cite book|last1=Coombs|first1=C. H.|last2=Dawes|first2=R. M.|last3=Tversky|first3=A.|year=1970|title=Mathematical Psychology: An Elementary Introduction|publisher=Prentice-Hall|location=Englewood Cliffs, New Jersey}}
* {{cite journal|first1=Kenneth Ward|last1=Church|first2=Patrick|last2=Hanks|title=Word association norms, mutual information, and lexicography|journal=Proceedings of the 27th Annual Meeting of the Association for Computational Linguistics|pages=76–83|year=1989|volume=16|issue=1|doi=10.3115/981623.981633|doi-access=free}}
* {{cite journal|first1=I.M.|last1=Gel'fand|first2=A.M.|last2=Yaglom|year=1957|title=Calculation of amount of information about a random function contained in another such function|journal= American Mathematical Society Translations |series=Series 2 |volume = 12 | pages = 199–246 |doi=10.1090/trans2/012/09|isbn=9780821817124}} English translation of original in ''Uspekhi Matematicheskikh Nauk'' '''12'''&nbsp;(1):&nbsp;3-52.
* {{cite book|last=Guiasu|first=Silviu|year=1977|title=Information Theory with Applications|publisher=McGraw-Hill, New York|isbn=978-0-07-025109-0}}
* {{cite book
  | last1 = Li
  | first1 = Ming
  | first2 = Paul
  | last2 = Vitányi
  | title = An introduction to Kolmogorov complexity and its applications
  | location = New York
  | publisher = [[Springer-Verlag]]
  | date=February 1997
  | isbn = 978-0-387-94868-3
  }}
* {{cite journal | last1 = Lockhead | first1 = G. R. | year = 1970 | title = Identification and the form of multidimensional discrimination space | journal = Journal of Experimental Psychology | volume = 85 | issue = 1| pages = 1–10 | doi=10.1037/h0029508| pmid = 5458322 }}
* David J. C. MacKay. ''[http://www.inference.phy.cam.ac.uk/mackay/itila/book.html Information Theory, Inference, and Learning Algorithms]'' Cambridge: Cambridge University Press, 2003. {{isbn|0-521-64298-1}} (available free online)
* {{cite journal | last1 = Haghighat | first1 = M. B. A. | last2 = Aghagolzadeh | first2 = A. | last3 = Seyedarabi | first3 = H. | year = 2011 | title = A non-reference image fusion metric based on mutual information of image features | journal = Computers & Electrical Engineering | volume = 37 | issue = 5| pages = 744–756 | doi=10.1016/j.compeleceng.2011.07.012| s2cid = 7738541 }}
* [[Athanasios Papoulis]]. ''Probability, Random Variables, and Stochastic Processes'', second edition. New York: McGraw-Hill, 1984. ''(See Chapter 15.)''
* {{cite book|last1=Witten|first1=Ian H.|last2=Frank|first2=Eibe |name-list-style=amp |year=2005|title=Data Mining: Practical Machine Learning Tools and Techniques|publisher=Morgan Kaufmann, Amsterdam|isbn=978-0-12-374856-0|url=http://www.cs.waikato.ac.nz/~ml/weka/book.html}}
* {{cite journal|author=Peng, H.C. |author2=Long, F. |author3=Ding, C. |name-list-style=amp |title=Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|volume=27|issue=8|pages=1226–1238|year=2005|url=http://research.janelia.org/peng/proj/mRMR/index.htm|doi=10.1109/tpami.2005.159|pmid=16119262|citeseerx=10.1.1.63.5765|s2cid=206764015 }}
* {{cite journal|author1=Andre S. Ribeiro |author2=Stuart A. Kauffman |author3=Jason Lloyd-Price |author4=Bjorn Samuelsson |author5=Joshua Socolar  |name-list-style=amp |year=2008|title=Mutual Information in Random Boolean models of regulatory networks|journal=Physical Review E|volume=77|issue=1|pages=011901 |arxiv=0707.3642|doi=10.1103/physreve.77.011901 |pmid=18351870 |bibcode=2008PhRvE..77a1901R|s2cid=15232112 }}
* {{cite journal
  | last1 = Wells
  | first1 = W.M. III
  | last2 = Viola
  | first2 = P.
  | last3 = Atsumi
  | first3 = H.
  | last4 = Nakajima
  | first4 = S.
  | last5 = Kikinis
  | first5 = R.
  | title = Multi-modal volume registration by maximization of mutual information
  | journal = Medical Image Analysis
  | volume = 1
  | issue = 1
  | pages = 35–51
  | year = 1996
  | pmid = 9873920
  | doi = 10.1016/S1361-8415(01)80004-9
  | url = http://www.ai.mit.edu/people/sw/papers/mia.pdf
  | access-date = 2010-08-05
  | archive-url = https://web.archive.org/web/20080906201633/http://www.ai.mit.edu/people/sw/papers/mia.pdf
  | archive-date = 2008-09-06
  | url-status = dead
  }}
* {{cite journal | last1 = Pandey | first1 = Biswajit | last2 = Sarkar | first2 = Suman | year = 2017 | title = How much a galaxy knows about its large-scale environment?: An information theoretic perspective | journal = Monthly Notices of the Royal Astronomical Society Letters| volume = 467 | issue = 1| page = L6 | doi=10.1093/mnrasl/slw250| doi-access = free | arxiv = 1611.00283| bibcode = 2017MNRAS.467L...6P| s2cid = 119095496 }}

* {{cite journal | last1 = Janssen | first1 = Joseph | last2 = Guan | first2 = Vincent | last3 = Robeva | first3 = Elina | year = 2023 | title = Ultra-marginal Feature Importance: Learning from Data with Causal Guarantees | journal = International Conference on Artificial Intelligence and Statistics| pages = 10782–10814 | arxiv = 2204.09938 | url= https://proceedings.mlr.press/v206/janssen23a.html}}

{{Authority control}}

[[Category:Information theory]]
[[Category:Entropy and information]]