Editing Marginal likelihood (section)

==Concept==
Given a set of [[independent identically distributed]] data points <math>\mathbf{X}=(x_1,\ldots,x_n),</math> where <math>x_i \sim p(x|\theta)</math> according to some [[probability distribution]] parameterized by <math>\theta</math>, where <math>\theta</math> itself is a [[random variable]] described by a distribution, i.e. <math>\theta \sim p(\theta\mid\alpha),</math> the marginal likelihood in general asks what the probability <math>p(\mathbf{X}\mid\alpha)</math> is, where <math>\theta</math> has been [[marginal distribution|marginalized out]] (integrated out):

:<math>p(\mathbf{X}\mid\alpha) = \int_\theta p(\mathbf{X}\mid\theta) \, p(\theta\mid\alpha)\ \operatorname{d}\!\theta </math>

The above definition is phrased in the context of [[Bayesian statistics]] in which case <math>p(\theta\mid\alpha)</math> is called prior density and <math>p(\mathbf{X}\mid\theta)</math> is the likelihood. Recognizing that the marginal likelihood is the normalizing constant of the Bayesian posterior density <math>p(\theta\mid\mathbf{X},\alpha)</math>, one also has the alternative expression<ref>{{cite journal |first=Siddhartha |last=Chib |title=Marginal likelihood from the Gibbs output |journal=Journal of the American Statistical Association |year=1995 |volume=90 |issue=432 |pages=1313–1321 |doi=10.1080/01621459.1995.10476635 }}</ref> 

:<math>p(\mathbf{X} \mid \alpha) = \frac{p(\mathbf{X} \mid \theta, \alpha) p(\theta \mid \alpha)}{p(\theta \mid \mathbf{X}, \alpha)}</math>

which is an identity in <math>\theta</math>. The marginal likelihood quantifies the agreement between data and prior in a geometric sense made precise{{How|date=February 2023}} in de Carvalho et al. (2019). In classical ([[frequentist statistics|frequentist]]) statistics, the concept of marginal likelihood occurs instead in the context of a joint parameter <math>\theta = (\psi,\lambda)</math>, where <math>\psi</math> is the actual parameter of interest, and <math>\lambda</math> is a non-interesting [[nuisance parameter]].  If there exists a probability distribution for <math>\lambda</math>{{Dubious|1=Frequentist_marginal_likelihood|reason=Parameters do not have distributions in frequentists statistics|date=February 2023}}, it is often desirable to consider the likelihood function only in terms of <math>\psi</math>, by marginalizing out <math>\lambda</math>:
:<math>\mathcal{L}(\psi;\mathbf{X}) = p(\mathbf{X}\mid\psi) = \int_\lambda p(\mathbf{X}\mid\lambda,\psi) \, p(\lambda\mid\psi) \ \operatorname{d}\!\lambda </math>

Unfortunately, marginal likelihoods are generally difficult to compute. Exact solutions are known for a small class of distributions, particularly when the marginalized-out parameter is the [[conjugate prior]] of the distribution of the data. In other cases, some kind of [[numerical integration]] method is needed, either a general method such as [[Gaussian integration]] or a [[Monte Carlo method]], or a method specialized to statistical problems such as the [[Laplace approximation]], [[Gibbs sampling|Gibbs]]/[[Metropolis–Hastings_algorithm|Metropolis]] sampling, or the [[EM algorithm]].

It is also possible to apply the above considerations to a single random variable (data point) <math>x</math>, rather than a set of observations.  In a Bayesian context, this is equivalent to the [[prior predictive distribution]] of a data point.