Editing Estimation theory (section)

==Examples==

===Unknown constant in additive white Gaussian noise===
Consider a received [[discrete signal]], <math>x[n]</math>, of <math>N</math> [[statistical independence|independent]] [[statistical sample|samples]] that consists of an unknown constant <math>A</math> with [[additive white Gaussian noise]] (AWGN) <math>w[n]</math> with zero [[mean]] and known [[variance]] <math>\sigma^2</math> (''i.e.'', <math>\mathcal{N}(0, \sigma^2)</math>).
Since the variance is known then the only unknown parameter is <math>A</math>.

The model for the signal is then
<math display="block">x[n] = A + w[n] \quad n=0, 1, \dots, N-1</math>

Two possible (of many) estimators for the parameter <math>A</math> are:
* <math>\hat{A}_1 = x[0]</math>
* <math>\hat{A}_2 = \frac{1}{N} \sum_{n=0}^{N-1} x[n]</math> which is the [[sample mean]]

Both of these estimators have a [[mean]] of <math>A</math>, which can be shown through taking the [[expected value]] of each estimator
<math display="block">\mathrm{E}\left[\hat{A}_1\right] = \mathrm{E}\left[ x[0] \right] = A</math>
and
<math display="block">
\mathrm{E}\left[ \hat{A}_2 \right]
=
\mathrm{E}\left[ \frac{1}{N} \sum_{n=0}^{N-1} x[n] \right]
=
\frac{1}{N} \left[ \sum_{n=0}^{N-1} \mathrm{E}\left[ x[n] \right] \right]
=
\frac{1}{N} \left[ N A \right]
=
A
</math>

At this point, these two estimators would appear to perform the same.
However, the difference between them becomes apparent when comparing the variances.
<math display="block">\mathrm{var} \left( \hat{A}_1 \right) = \mathrm{var} \left( x[0] \right) = \sigma^2</math>
and
<math display="block">
\mathrm{var} \left( \hat{A}_2 \right)
=
\mathrm{var} \left( \frac{1}{N} \sum_{n=0}^{N-1} x[n] \right)
\overset{\text{independence}}{=}
\frac{1}{N^2} \left[ \sum_{n=0}^{N-1} \mathrm{var} (x[n]) \right]
=
\frac{1}{N^2} \left[ N \sigma^2 \right]
=
\frac{\sigma^2}{N}
</math>

It would seem that the sample mean is a better estimator since its variance is lower for every&nbsp;''N''&nbsp;>&nbsp;1.

====Maximum likelihood====
{{main|Maximum likelihood}}

Continuing the example using the [[maximum likelihood]] estimator, the [[probability density function]] (pdf) of the noise for one sample <math>w[n]</math> is
<math display="block">p(w[n]) = \frac{1}{\sigma \sqrt{2 \pi}} \exp\left(- \frac{1}{2 \sigma^2} w[n]^2 \right)</math>
and the probability of <math>x[n]</math> becomes (<math>x[n]</math> can be thought of a <math>\mathcal{N}(A, \sigma^2)</math>)
<math display="block">p(x[n]; A) = \frac{1}{\sigma \sqrt{2 \pi}} \exp\left(- \frac{1}{2 \sigma^2} (x[n] - A)^2 \right)</math>
By [[statistical independence|independence]], the probability of <math>\mathbf{x}</math> becomes
<math display="block">
p(\mathbf{x}; A)
=
\prod_{n=0}^{N-1} p(x[n]; A)
=
\frac{1}{\left(\sigma \sqrt{2\pi}\right)^N}
\exp\left(- \frac{1}{2 \sigma^2} \sum_{n=0}^{N-1}(x[n] - A)^2 \right)
</math>
Taking the [[natural logarithm]] of the pdf
<math display="block">
\ln p(\mathbf{x}; A)
=
-N \ln \left(\sigma \sqrt{2\pi}\right)
- \frac{1}{2 \sigma^2} \sum_{n=0}^{N-1}(x[n] - A)^2
</math>
and the maximum likelihood estimator is
<math display="block">\hat{A} = \arg \max \ln p(\mathbf{x}; A)</math>

Taking the first [[derivative]] of the log-likelihood function
<math display="block">
\frac{\partial}{\partial A} \ln p(\mathbf{x}; A)
=
\frac{1}{\sigma^2} \left[ \sum_{n=0}^{N-1}(x[n] - A) \right]
=
\frac{1}{\sigma^2} \left[ \sum_{n=0}^{N-1}x[n] - N A \right]
</math>
and setting it to zero
<math display="block">
0
=
\frac{1}{\sigma^2} \left[ \sum_{n=0}^{N-1}x[n] - N A \right]
=
\sum_{n=0}^{N-1}x[n] - N A
</math>

This results in the maximum likelihood estimator
<math display="block">\hat{A} = \frac{1}{N} \sum_{n=0}^{N-1}x[n]</math>
which is simply the sample mean.
From this example, it was found that the sample mean is the maximum likelihood estimator for <math>N</math> samples of a fixed, unknown parameter corrupted by AWGN.

====Cramér–Rao lower bound====
{{further|Cramér–Rao bound}}

To find the [[Cramér–Rao lower bound]] (CRLB) of the sample mean estimator, it is first necessary to find the [[Fisher information]] number
<math display="block">
\mathcal{I}(A)
=
\mathrm{E}
\left(
 \left[
  \frac{\partial}{\partial A} \ln p(\mathbf{x}; A)
 \right]^2
\right)
=
-\mathrm{E}
\left[
 \frac{\partial^2}{\partial A^2} \ln p(\mathbf{x}; A)
\right]
</math>
and copying from above
<math display="block">
\frac{\partial}{\partial A} \ln p(\mathbf{x}; A)
=
\frac{1}{\sigma^2} \left[ \sum_{n=0}^{N-1}x[n] - N A \right]
</math>

Taking the second derivative
<math display="block">
\frac{\partial^2}{\partial A^2} \ln p(\mathbf{x}; A)
=
\frac{1}{\sigma^2} (- N)
=
\frac{-N}{\sigma^2}
</math>
and finding the negative expected value is trivial since it is now a deterministic constant
<math>
-\mathrm{E}
\left[
 \frac{\partial^2}{\partial A^2} \ln p(\mathbf{x}; A)
\right]
=
\frac{N}{\sigma^2}
</math>

Finally, putting the Fisher information into
<math display="block">
\mathrm{var}\left( \hat{A} \right)
\geq
\frac{1}{\mathcal{I}}
</math>
results in
<math display="block">
\mathrm{var}\left( \hat{A} \right)
\geq
\frac{\sigma^2}{N}
</math>

Comparing this to the variance of the sample mean (determined previously) shows that the sample mean is ''equal to'' the Cramér–Rao lower bound for all values of <math>N</math> and <math>A</math>.
In other words, the sample mean is the (necessarily unique) [[efficient estimator]], and thus also the [[minimum variance unbiased estimator]] (MVUE), in addition to being the [[maximum likelihood]] estimator.

===Maximum of a uniform distribution===
{{main|German tank problem}}

One of the simplest non-trivial examples of estimation is the estimation of the maximum of a uniform distribution. It is used as a hands-on classroom exercise and to illustrate basic principles of estimation theory. Further, in the case of estimation based on a single sample, it demonstrates philosophical issues and possible misunderstandings in the use of [[maximum likelihood]] estimators and [[likelihood functions]].

Given a [[discrete uniform distribution]] <math>1,2,\dots,N</math> with unknown maximum, the [[UMVU]] estimator for the maximum is given by
<math display="block">\frac{k+1}{k} m - 1 = m + \frac{m}{k} - 1</math>
where ''m'' is the [[sample maximum]] and ''k'' is the [[sample size]], sampling without replacement.<ref name="Johnson">{{citation | last=Johnson | first=Roger | title=Estimating the Size of a Population | year=1994 | journal=Teaching Statistics | volume=16 | issue=2 (Summer) | doi=10.1111/j.1467-9639.1994.tb00688.x | pages = 50–52 }}</ref><ref name="Johnson2">{{citation | last=Johnson | first=Roger | contribution=Estimating the Size of a Population | title=Getting the Best from Teaching Statistics | year=2006 | url=http://www.rsscse.org.uk/ts/gtb/contents.html | contribution-url=http://www.rsscse.org.uk/ts/gtb/johnson.pdf | url-status=dead | archive-url=https://web.archive.org/web/20081120085633/http://www.rsscse.org.uk/ts/gtb/contents.html | archive-date=November 20, 2008 }}</ref> This problem is commonly known as the [[German tank problem]], due to application of maximum estimation to estimates of German tank production during [[World War II]].

The formula may be understood intuitively as;
{{block indent | em = 1.5 | text = "The sample maximum plus the average gap between observations in the sample",}}
the gap being added to compensate for the negative bias of the sample maximum as an estimator for the population maximum.{{NoteTag|The sample maximum is never more than the population maximum, but can be less, hence it is a [[biased estimator]]: it will tend to ''underestimate'' the population maximum.}}

This has a variance of<ref name="Johnson" />
<math display="block">\frac{1}{k}\frac{(N-k)(N+1)}{(k+2)} \approx \frac{N^2}{k^2} \text{ for small samples } k \ll N</math>
so a standard deviation of approximately <math>N/k</math>, the (population) average size of a gap between samples; compare <math>\frac{m}{k}</math> above. This can be seen as a very simple case of [[maximum spacing estimation]].

The sample maximum is the [[maximum likelihood]] estimator for the population maximum, but, as discussed above, it is biased.