Editing Estimation theory (section)

===Unknown constant in additive white Gaussian noise===
Consider a received [[discrete signal]], <math>x[n]</math>, of <math>N</math> [[statistical independence|independent]] [[statistical sample|samples]] that consists of an unknown constant <math>A</math> with [[additive white Gaussian noise]] (AWGN) <math>w[n]</math> with zero [[mean]] and known [[variance]] <math>\sigma^2</math> (''i.e.'', <math>\mathcal{N}(0, \sigma^2)</math>).
Since the variance is known then the only unknown parameter is <math>A</math>.

The model for the signal is then
<math display="block">x[n] = A + w[n] \quad n=0, 1, \dots, N-1</math>

Two possible (of many) estimators for the parameter <math>A</math> are:
* <math>\hat{A}_1 = x[0]</math>
* <math>\hat{A}_2 = \frac{1}{N} \sum_{n=0}^{N-1} x[n]</math> which is the [[sample mean]]

Both of these estimators have a [[mean]] of <math>A</math>, which can be shown through taking the [[expected value]] of each estimator
<math display="block">\mathrm{E}\left[\hat{A}_1\right] = \mathrm{E}\left[ x[0] \right] = A</math>
and
<math display="block">
\mathrm{E}\left[ \hat{A}_2 \right]
=
\mathrm{E}\left[ \frac{1}{N} \sum_{n=0}^{N-1} x[n] \right]
=
\frac{1}{N} \left[ \sum_{n=0}^{N-1} \mathrm{E}\left[ x[n] \right] \right]
=
\frac{1}{N} \left[ N A \right]
=
A
</math>

At this point, these two estimators would appear to perform the same.
However, the difference between them becomes apparent when comparing the variances.
<math display="block">\mathrm{var} \left( \hat{A}_1 \right) = \mathrm{var} \left( x[0] \right) = \sigma^2</math>
and
<math display="block">
\mathrm{var} \left( \hat{A}_2 \right)
=
\mathrm{var} \left( \frac{1}{N} \sum_{n=0}^{N-1} x[n] \right)
\overset{\text{independence}}{=}
\frac{1}{N^2} \left[ \sum_{n=0}^{N-1} \mathrm{var} (x[n]) \right]
=
\frac{1}{N^2} \left[ N \sigma^2 \right]
=
\frac{\sigma^2}{N}
</math>

It would seem that the sample mean is a better estimator since its variance is lower for every&nbsp;''N''&nbsp;>&nbsp;1.

====Maximum likelihood====
{{main|Maximum likelihood}}

Continuing the example using the [[maximum likelihood]] estimator, the [[probability density function]] (pdf) of the noise for one sample <math>w[n]</math> is
<math display="block">p(w[n]) = \frac{1}{\sigma \sqrt{2 \pi}} \exp\left(- \frac{1}{2 \sigma^2} w[n]^2 \right)</math>
and the probability of <math>x[n]</math> becomes (<math>x[n]</math> can be thought of a <math>\mathcal{N}(A, \sigma^2)</math>)
<math display="block">p(x[n]; A) = \frac{1}{\sigma \sqrt{2 \pi}} \exp\left(- \frac{1}{2 \sigma^2} (x[n] - A)^2 \right)</math>
By [[statistical independence|independence]], the probability of <math>\mathbf{x}</math> becomes
<math display="block">
p(\mathbf{x}; A)
=
\prod_{n=0}^{N-1} p(x[n]; A)
=
\frac{1}{\left(\sigma \sqrt{2\pi}\right)^N}
\exp\left(- \frac{1}{2 \sigma^2} \sum_{n=0}^{N-1}(x[n] - A)^2 \right)
</math>
Taking the [[natural logarithm]] of the pdf
<math display="block">
\ln p(\mathbf{x}; A)
=
-N \ln \left(\sigma \sqrt{2\pi}\right)
- \frac{1}{2 \sigma^2} \sum_{n=0}^{N-1}(x[n] - A)^2
</math>
and the maximum likelihood estimator is
<math display="block">\hat{A} = \arg \max \ln p(\mathbf{x}; A)</math>

Taking the first [[derivative]] of the log-likelihood function
<math display="block">
\frac{\partial}{\partial A} \ln p(\mathbf{x}; A)
=
\frac{1}{\sigma^2} \left[ \sum_{n=0}^{N-1}(x[n] - A) \right]
=
\frac{1}{\sigma^2} \left[ \sum_{n=0}^{N-1}x[n] - N A \right]
</math>
and setting it to zero
<math display="block">
0
=
\frac{1}{\sigma^2} \left[ \sum_{n=0}^{N-1}x[n] - N A \right]
=
\sum_{n=0}^{N-1}x[n] - N A
</math>

This results in the maximum likelihood estimator
<math display="block">\hat{A} = \frac{1}{N} \sum_{n=0}^{N-1}x[n]</math>
which is simply the sample mean.
From this example, it was found that the sample mean is the maximum likelihood estimator for <math>N</math> samples of a fixed, unknown parameter corrupted by AWGN.

====Cramér–Rao lower bound====
{{further|Cramér–Rao bound}}

To find the [[Cramér–Rao lower bound]] (CRLB) of the sample mean estimator, it is first necessary to find the [[Fisher information]] number
<math display="block">
\mathcal{I}(A)
=
\mathrm{E}
\left(
 \left[
  \frac{\partial}{\partial A} \ln p(\mathbf{x}; A)
 \right]^2
\right)
=
-\mathrm{E}
\left[
 \frac{\partial^2}{\partial A^2} \ln p(\mathbf{x}; A)
\right]
</math>
and copying from above
<math display="block">
\frac{\partial}{\partial A} \ln p(\mathbf{x}; A)
=
\frac{1}{\sigma^2} \left[ \sum_{n=0}^{N-1}x[n] - N A \right]
</math>

Taking the second derivative
<math display="block">
\frac{\partial^2}{\partial A^2} \ln p(\mathbf{x}; A)
=
\frac{1}{\sigma^2} (- N)
=
\frac{-N}{\sigma^2}
</math>
and finding the negative expected value is trivial since it is now a deterministic constant
<math>
-\mathrm{E}
\left[
 \frac{\partial^2}{\partial A^2} \ln p(\mathbf{x}; A)
\right]
=
\frac{N}{\sigma^2}
</math>

Finally, putting the Fisher information into
<math display="block">
\mathrm{var}\left( \hat{A} \right)
\geq
\frac{1}{\mathcal{I}}
</math>
results in
<math display="block">
\mathrm{var}\left( \hat{A} \right)
\geq
\frac{\sigma^2}{N}
</math>

Comparing this to the variance of the sample mean (determined previously) shows that the sample mean is ''equal to'' the Cramér–Rao lower bound for all values of <math>N</math> and <math>A</math>.
In other words, the sample mean is the (necessarily unique) [[efficient estimator]], and thus also the [[minimum variance unbiased estimator]] (MVUE), in addition to being the [[maximum likelihood]] estimator.