Editing Least squares (section)

===Linear least squares===
{{main|Linear least squares (mathematics)|l1=Linear least squares}}

A regression model is a linear one when the model comprises a [[linear combination]] of the parameters, i.e.,
<math display="block"> f(x, \boldsymbol \beta) = \sum_{j = 1}^m \beta_j \phi_j(x),</math>
where the function <math>\phi_j</math> is a function of <math> x </math>.<ref name=":1" />

Letting <math> X_{ij}= \phi_j(x_{i})</math> and putting the independent and dependent variables in matrices <math> X</math> and <math> Y,</math> respectively, we can compute the least squares in the following way. Note that <math> D</math> is the set of all data.<ref name=":1" /><ref name=":2">{{Cite book|last1=Rencher|first1=Alvin C.|url=https://books.google.com/books?id=0g-PAuKub3QC&pg=PA19|title=Methods of Multivariate Analysis|last2=Christensen|first2=William F.|date=2012-08-15|publisher=John Wiley & Sons|isbn=978-1-118-39167-9|pages=155|language=en}}</ref>
<math display="block">L(D, \boldsymbol{\beta}) = \left\|Y - X\boldsymbol{\beta} \right\|^2
= (Y - X\boldsymbol{\beta})^\mathsf{T} (Y - X\boldsymbol{\beta})</math>
<math display="block">= Y^\mathsf{T}Y- 2Y^\mathsf{T}X\boldsymbol{\beta} + \boldsymbol{\beta}^\mathsf{T}X^\mathsf{T}X\boldsymbol{\beta}
</math>

The gradient of the loss is:
<math display="block">\frac{\partial L(D, \boldsymbol{\beta})}{\partial \boldsymbol{\beta}}
= \frac{\partial \left(Y^\mathsf{T}Y- 2Y^\mathsf{T}X\boldsymbol{\beta} + \boldsymbol{\beta}^\mathsf{T}X^\mathsf{T}X\boldsymbol{\beta}\right)}{\partial \boldsymbol{\beta}}
= -2X^\mathsf{T}Y + 2X^\mathsf{T}X\boldsymbol{\beta}</math>

Setting the gradient of the loss to zero and solving for <math>\boldsymbol{\beta}</math>, we get:<ref name=":2" /><ref name=":1" /> 
<math display="block">-2X^\mathsf{T}Y + 2X^\mathsf{T}X\boldsymbol{\beta} = 0
\Rightarrow X^\mathsf{T}Y = X^\mathsf{T}X\boldsymbol{\beta}</math>
<math display="block">\boldsymbol{\hat{\beta}} = \left(X^\mathsf{T}X\right)^{-1} X^\mathsf{T}Y</math>