Editing Mixture model (section)

====Categorical mixture model====
[[File:nonbayesian-categorical-mixture.svg|right|250px|thumb|Non-Bayesian categorical mixture model using [[plate notation]].  Smaller squares indicate fixed parameters; larger circles indicate random variables.  Filled-in shapes indicate known values.  The indication [K] means a vector of size ''K''; likewise for [V].]]

A typical non-Bayesian mixture model with [[categorical distribution|categorical]] observations looks like this:

*<math>K,N:</math> as above
*<math>\phi_{i=1 \dots K}, \boldsymbol\phi:</math> as above
*<math>z_{i=1 \dots N}, x_{i=1 \dots N}:</math> as above
*<math>V:</math> dimension of categorical observations, e.g., size of word vocabulary
*<math>\theta_{i=1 \dots K, j=1 \dots V}:</math> probability for component <math>i</math> of observing item <math>j</math>
*<math>\boldsymbol\theta_{i=1 \dots K}:</math> vector of dimension <math>V,</math> composed of <math>\theta_{i,1 \dots V};</math> must sum to 1

The random variables:
:<math>
\begin{array}{lcl}
z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\
x_{i=1 \dots N} &\sim& \text{Categorical}(\boldsymbol\theta_{z_i})
\end{array}
</math>

<!--
The original version, all in LaTeX.
:<math>
\begin{array}{lcl}
K,N &=& \text{as above} \\
\phi_{i=1 \dots K}, \boldsymbol\phi &=& \text{as above} \\
z_{i=1 \dots N}, x_{i=1 \dots N} &=& \text{as above} \\
V &=& \text{dimension of categorical observations, e.g., size of word vocabulary} \\
\theta_{i=1 \dots K, j=1 \dots V} &=& \text{probability for component } i \text{ of observing the } j\text{th item} \\
\boldsymbol\theta_{i=1 \dots K} &=& V\text{-dimensional vector, composed of }\theta_{i,1 \dots V} \text{; must sum to 1} \\
z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\
x_{i=1 \dots N} &\sim& \text{Categorical}(\boldsymbol\theta_{z_i})
\end{array}
</math>
-->

{{clear}}
[[File:bayesian-categorical-mixture.svg|right|300px|thumb|Bayesian categorical mixture model using [[plate notation]].  Smaller squares indicate fixed parameters; larger circles indicate random variables.  Filled-in shapes indicate known values.  The indication [K] means a vector of size ''K''; likewise for [V].]]

A typical Bayesian mixture model with [[categorical distribution|categorical]] observations looks like this:

*<math>K,N:</math> as above
*<math>\phi_{i=1 \dots K}, \boldsymbol\phi:</math> as above
*<math>z_{i=1 \dots N}, x_{i=1 \dots N}:</math> as above
*<math>V:</math> dimension of categorical observations, e.g., size of word vocabulary
*<math>\theta_{i=1 \dots K, j=1 \dots V}:</math> probability for component <math>i</math> of observing item <math>j</math>
*<math>\boldsymbol\theta_{i=1 \dots K}:</math> vector of dimension <math>V,</math> composed of <math>\theta_{i,1 \dots V};</math> must sum to 1
*<math>\alpha:</math> shared concentration hyperparameter of <math>\boldsymbol\theta</math> for each component
*<math>\beta:</math> concentration hyperparameter of <math>\boldsymbol\phi</math>

The random variables:
:<math>
\begin{array}{lcl}
\boldsymbol\phi &\sim& \operatorname{Symmetric-Dirichlet}_K(\beta) \\
\boldsymbol\theta_{i=1 \dots K} &\sim& \text{Symmetric-Dirichlet}_V(\alpha) \\
z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\
x_{i=1 \dots N} &\sim& \text{Categorical}(\boldsymbol\theta_{z_i})
\end{array}
</math>

<!--
The (beginning of) equivalent of below, using no LaTeX.

*''K'',''N'' = as above
*&phi;<sub>1,...,''K''</sub>, '''&phi;''' as above
*''z''<sub>''i''=1...''N''</sub>, ''x''<sub>''i''=1...''N''</sub> = as above
* ''V'' = dimension of categorical observations, e.g., size of word vocabulary
-->
<!--
The equivalent using full LaTeX.

:<math>
\begin{array}{lcl}
K,N &=& \mbox{as above} \\
\phi_{i=1 \dots K}, \boldsymbol\phi &=& \text{as above} \\
z_{i=1 \dots N}, x_{i=1 \dots N} &=& \text{as above} \\
V &=& \text{dimension of categorical observations, e.g., size of word vocabulary} \\
\theta_{i=1 \dots K, j=1 \dots V} &=& \text{probability for component } i \text{ of observing the } j\text{th item} \\
\boldsymbol\theta_{i=1 \dots K} &=& V\text{-dimensional vector, composed of }\theta_{i,1 \dots V} \text{; must sum to 1} \\
\alpha &=& \text{shared concentration hyperparameter of } \boldsymbol\theta \text{ for each component} \\
\beta &=& \text{concentration hyperparameter of } \boldsymbol\phi \\
\boldsymbol\phi &\sim& \operatorname{Symmetric-Dirichlet}_K(\beta) \\
\boldsymbol\theta_{i=1 \dots K} &\sim& \text{Symmetric-Dirichlet}_V(\alpha) \\
z_{i=1 \dots N} &\sim& \operatorname{Categorical}(\boldsymbol\phi) \\
x_{i=1 \dots N} &\sim& \text{Categorical}(\boldsymbol\theta_{z_i})
\end{array}
</math>
-->