Editing Autoregressive model (section)

==Calculation of the AR parameters==

There are many ways to estimate the coefficients, such as the [[ordinary least squares]] procedure or [[Method of moments (statistics)|method of moments]] (through Yule–Walker equations).

The AR(''p'') model is given by the equation

:<math> X_t = \sum_{i=1}^p \varphi_i X_{t-i}+ \varepsilon_t.\,</math>

It is based on parameters <math>\varphi_i</math> where ''i'' = 1, ..., ''p''. There is a direct correspondence between these parameters and the covariance function of the process, and this correspondence can be inverted to determine the parameters from the autocorrelation function (which is itself obtained from the covariances). This is done using the Yule–Walker equations.

===Yule–Walker equations===
<!-- this heading is linked from other articles -->

The Yule–Walker equations, named for [[Udny Yule]] and [[Gilbert Walker (physicist)|Gilbert Walker]],<ref>Yule, G. Udny (1927) [http://visualiseur.bnf.fr/Visualiseur?Destination=Gallica&O=NUMM-56031 "On a Method of Investigating Periodicities in Disturbed Series, with Special Reference to Wolfer's Sunspot Numbers"] {{Webarchive|url=https://web.archive.org/web/20110514094546/http://visualiseur.bnf.fr/Visualiseur?Destination=Gallica&O=NUMM-56031 |date=2011-05-14 }}, ''[[Philosophical Transactions of the Royal Society]] of London'', Ser. A, Vol. 226, 267–298.]</ref><ref>Walker, Gilbert  (1931) [http://visualiseur.bnf.fr/Visualiseur?Destination=Gallica&O=NUMM-56224  "On Periodicity in Series of Related Terms"] {{Webarchive|url=https://web.archive.org/web/20110607170511/http://visualiseur.bnf.fr/Visualiseur?Destination=Gallica&O=NUMM-56224 |date=2011-06-07 }}, ''[[Proceedings of the Royal Society]] of London'', Ser. A, Vol. 131,  518–532.</ref> are the following set of equations.<ref>{{cite book |last=Theodoridis |first=Sergios |title=Machine Learning: A Bayesian and Optimization Perspective |publisher=Academic Press, 2015 |chapter=Chapter 1. Probability and Stochastic Processes |pages=9–51 |isbn=978-0-12-801522-3 |date=2015-04-10 }}</ref>

:<math>\gamma_m = \sum_{k=1}^p \varphi_k \gamma_{m-k} + \sigma_\varepsilon^2\delta_{m,0},</math>

where {{nowrap|1=''m'' = 0, …, ''p''}}, yielding {{nowrap|''p'' + 1}} equations. Here <math>\gamma_m</math> is the autocovariance function of X<sub>t</sub>, <math>\sigma_\varepsilon</math> is the standard deviation of the input noise process, and <math>\delta_{m,0}</math> is the [[Kronecker delta function]].

Because the last part of an individual equation is non-zero only if {{nowrap|1=''m'' = 0}}, the set of equations can be solved by representing the equations for {{nowrap|''m'' > 0}} in matrix form, thus getting the equation

:<math>\begin{bmatrix}
\gamma_1 \\
\gamma_2 \\
\gamma_3 \\
\vdots \\
\gamma_p \\
\end{bmatrix}

=

\begin{bmatrix}
\gamma_0     & \gamma_{-1}  & \gamma_{-2}  & \cdots \\
\gamma_1     & \gamma_0     & \gamma_{-1}  & \cdots \\
\gamma_2     & \gamma_1     & \gamma_0     & \cdots \\
\vdots       & \vdots       & \vdots       & \ddots \\
\gamma_{p-1} & \gamma_{p-2} & \gamma_{p-3} & \cdots \\
\end{bmatrix}

\begin{bmatrix}
\varphi_{1} \\
\varphi_{2} \\
\varphi_{3} \\
 \vdots \\
\varphi_{p} \\
\end{bmatrix}

</math>

which can be solved for all <math>\{\varphi_m; m=1,2, \dots ,p\}.</math> The remaining equation for ''m'' = 0 is

:<math>\gamma_0 = \sum_{k=1}^p \varphi_k \gamma_{-k} + \sigma_\varepsilon^2 ,</math>

which, once  <math>\{\varphi_m ; m=1,2, \dots ,p \}</math> are known, can be solved for <math>\sigma_\varepsilon^2 .</math>

An alternative formulation is in terms of the [[autocorrelation function]]. The AR parameters are determined by the first ''p''+1 elements <math>\rho(\tau)</math> of the autocorrelation function. The full autocorrelation function can then be derived by recursively calculating
<ref name=Storch>{{Cite book
| publisher = Cambridge University Press
| isbn = 0-521-01230-9
| last = Von Storch
| first = Hans
| first2=Francis W. |last2=Zwiers
| title = Statistical analysis in climate research
| year = 2001
| doi = 10.1017/CBO9780511612336
}}{{Page needed|date=March 2011}}</ref>
: <math>\rho(\tau) = \sum_{k=1}^p \varphi_k \rho(k-\tau)</math>
Examples for some Low-order AR(''p'') processes
* ''p''=1
** <math>\gamma_1 = \varphi_1 \gamma_0</math>
** Hence <math>\rho_1 = \gamma_1 / \gamma_0 = \varphi_1</math>
* ''p''=2
** The Yule–Walker equations for an AR(2) process are
**: <math>\gamma_1 = \varphi_1 \gamma_0 + \varphi_2 \gamma_{-1}</math>
**: <math>\gamma_2 = \varphi_1 \gamma_1 + \varphi_2 \gamma_0</math>
*** Remember that <math>\gamma_{-k} = \gamma_k</math>
*** Using the first equation yields <math>\rho_1 = \gamma_1 / \gamma_0 = \frac{\varphi_1}{1-\varphi_2}</math>
*** Using the recursion formula yields <math>\rho_2 = \gamma_2 / \gamma_0 = \frac{\varphi_1^2 - \varphi_2^2 + \varphi_2}{1 - \varphi_2}</math>

===Estimation of AR parameters===

The above equations (the Yule–Walker equations) provide several routes to estimating the parameters of an AR(''p'') model, by replacing the theoretical covariances with estimated values.<ref>{{Cite web |url=http://www-stat.wharton.upenn.edu/~steele/Courses/956/Resource/YWSourceFiles/YW-Eshel.pdf |title=The Yule Walker Equations for the AR Coefficients |last=Eshel |first=Gidon |website=stat.wharton.upenn.edu |access-date=2019-01-27 |archive-date=2018-07-13 |archive-url=https://web.archive.org/web/20180713135223/http://www-stat.wharton.upenn.edu/~steele/Courses/956/Resource/YWSourceFiles/YW-Eshel.pdf |url-status=live }}</ref> Some of these variants can be described as follows:

*Estimation of autocovariances or autocorrelations. Here each of these terms is estimated separately, using conventional estimates. There are different ways of doing this and the choice between these affects the properties of the estimation scheme. For example, negative estimates of the variance can be produced by some choices.
*Formulation as a [[least squares regression]] problem in which an ordinary least squares prediction problem is constructed, basing prediction of values of ''X''<sub>''t''</sub> on the ''p'' previous values of the same series. This can be thought of as a forward-prediction scheme. The [[normal equations]] for this problem can be seen to correspond to an approximation of the matrix form of the Yule–Walker equations in which each appearance of an autocovariance of the same lag is replaced by a slightly different estimate.
*Formulation as an extended form of ordinary least squares prediction problem. Here two sets of prediction equations are combined into a single estimation scheme and a single set of normal equations. One set is the set of forward-prediction equations and the other is a corresponding set of backward prediction equations, relating to the backward representation of the AR model:
::<math> X_t = \sum_{i=1}^p \varphi_i X_{t+i}+ \varepsilon^*_t \,.</math>
:Here predicted values of ''X''<sub>''t''</sub> would be based on the ''p'' future values of the same series.{{Clarify|reason=1) the latest equation was introduced as "the backward representation of the AR model"; 2) assuming t is counted in the same direction as the progression of time, then the subscripts t-i indicate past values of X w.r.t. X_t; 3) the last sentence would indicate that - suddenly - X_t would depend on future (rather than past) values of X|date=July 2020}} This way of estimating the AR parameters is due to John Parker Burg,<ref name=Burg/> and is called the Burg method:<ref name=Brockwell/> Burg and later authors called these particular estimates "maximum entropy estimates",<ref name=Burg1/> but the reasoning behind this applies to the use of any set of estimated AR parameters. Compared to the estimation scheme using only the forward prediction equations, different estimates of the autocovariances are produced, and the estimates have different stability properties. Burg estimates are particularly associated with [[maximum entropy spectral estimation]].<ref name=Bos/>

Other possible approaches to estimation include [[maximum likelihood estimation]]. Two distinct variants of maximum likelihood are available: in one (broadly equivalent to the forward prediction least squares scheme) the likelihood function considered is that corresponding to the conditional distribution of later values in the series given the initial ''p'' values in the series; in the second, the likelihood function considered is that corresponding to the unconditional joint distribution of all the values in the observed series. Substantial differences in the results of these approaches can occur if the observed series is short, or if the process is close to non-stationarity.