Open main menu
Home
Random
Recent changes
Special pages
Community portal
Preferences
About Wikipedia
Disclaimers
Incubator escapee wiki
Search
User menu
Talk
Dark mode
Contributions
Create account
Log in
Editing
Least squares
Warning:
You are not logged in. Your IP address will be publicly visible if you make any edits. If you
log in
or
create an account
, your edits will be attributed to your username, along with other benefits.
Anti-spam check. Do
not
fill this in!
{{short description|Approximation method in statistics}} {{distinguish-redirect|Least squares approximation|Least-squares function approximation}} {{Regression bar}} {{longlead|date=April 2025}} [[File:Linear least squares2.svg|right|thumb|The result of fitting a set of data points with a quadratic function]] [[File:X33-ellips-1.svg|thumb|Conic fitting a set of points using least-squares approximation]] In [[regression analysis]], '''least squares''' is a [[parameter estimation]] method in which the sum of the squares of the [[Residuals (statistics)|residuals]] (a residual being the difference between an observed value and the fitted value provided by a model) is minimized. The most important application is in [[curve fitting|data fitting]]. When the problem has substantial [[Uncertainty|uncertainties]] in the [[independent variable]] (the ''x'' variable), then simple regression and least-squares methods have problems; in such cases, the methodology required for fitting [[errors-in-variables models]] may be considered instead of that for least squares. Least squares problems fall into two categories: linear or [[ordinary least squares]] and [[nonlinear least squares]], depending on whether or not the model functions are linear in all unknowns. The linear least-squares problem occurs in statistical [[regression analysis]]; it has a [[closed-form solution]]. The nonlinear problem is usually solved by [[iterative refinement]]; at each iteration the system is approximated by a linear one, and thus the core calculation is similar in both cases. [[Polynomial least squares]] describes the [[variance]] in a prediction of the dependent variable as a function of the independent variable and the [[Deviation (statistics)|deviations]] from the fitted curve. When the observations come from an [[exponential family]] with identity as its natural sufficient statistics and mild-conditions are satisfied (e.g. for [[Normal distribution|normal]], [[Exponential distribution|exponential]], [[Poisson distribution|Poisson]] and [[Binomial distribution|binomial distributions]]), standardized least-squares estimates and [[Maximum likelihood|maximum-likelihood]] estimates are identical.<ref>{{Cite journal | last1 = Charnes | first1 = A. | last2 = Frome | first2 = E. L. | last3 = Yu | first3 = P. L. | doi = 10.1080/01621459.1976.10481508 | title = The Equivalence of Generalized Least Squares and Maximum Likelihood Estimates in the Exponential Family | journal = Journal of the American Statistical Association | volume = 71 | issue = 353 | pages = 169–171 | year = 1976 }}</ref> The method of least squares can also be derived as a [[method of moments (statistics)|method of moments]] estimator. The following discussion is mostly presented in terms of [[linear]] functions but the use of least squares is valid and practical for more general families of functions. Also, by iteratively applying local quadratic approximation to the likelihood (through the [[Fisher information]]), the least-squares method may be used to fit a [[generalized linear model]]. The least-squares method was officially discovered and published by [[Adrien-Marie Legendre]] (1805),<ref>Mansfield Merriman, "A List of Writings Relating to the Method of Least Squares"</ref> though it is usually also co-credited to [[Carl Friedrich Gauss]] (1809),<ref name=brertscher>{{cite book |last=Bretscher |first=Otto |title=Linear Algebra With Applications |edition=3rd |publisher=Prentice Hall |year=1995 |location=Upper Saddle River, NJ}}</ref><ref name=":5">{{cite journal |first=Stephen M. |last=Stigler |year=1981 |title=Gauss and the Invention of Least Squares |journal=Ann. Stat. |volume=9 |issue=3 |pages=465–474 |doi=10.1214/aos/1176345451 |doi-access=free }}</ref> who contributed significant theoretical advances to the method,<ref name=":5" /> and may have also used it in his earlier work in 1794 and 1795.<ref name=":3">{{Cite journal |last=Plackett |first=R.L. |author-link=R. L. Plackett |date=1972 |title=The discovery of the method of least squares |url=https://hedibert.org/wp-content/uploads/2016/08/plackett1972-thediscoveryofthemethodofleastsquares.pdf |journal=Biometrika |volume=59 |issue=2 |pages=239–251}}</ref><ref name=":5" /> ==History== ===Founding=== The method of least squares grew out of the fields of [[astronomy]] and [[geodesy]], as scientists and mathematicians sought to provide solutions to the challenges of navigating the Earth's oceans during the [[Age of Discovery]]. The accurate description of the behavior of celestial bodies was the key to enabling ships to sail in open seas, where sailors could no longer rely on land sightings for navigation. The method was the culmination of several advances that took place during the course of the eighteenth century:<ref name=stigler>{{cite book | last = Stigler | first = Stephen M. | title = The History of Statistics: The Measurement of Uncertainty Before 1900 | publisher = Belknap Press of Harvard University Press | year = 1986 | location = Cambridge, MA | isbn = 978-0-674-40340-6 | url-access = registration | url = https://archive.org/details/historyofstatist00stig }}</ref> *The combination of different observations as being the best estimate of the true value; errors decrease with aggregation rather than increase, first appeared in [[Isaac Newton]]'s work in 1671, though it went unpublished, and again in 1700.<ref>{{Cite book |last1=Buchwald |first1=Jed Z. |title=Newton and the Origin of Civilization |last2=Feingold |first2=Mordechai |date=2013 |publisher=Princeton University Press |isbn=978-0-691-15478-7 |location=Princeton Oxford |pages=90–93, 101–103}}</ref><ref>{{Cite web |last=Drum |first=Kevin |date=2013-05-10 |title=The Groundbreaking Isaac Newton Invention You've Never Heard Of |url=https://www.motherjones.com/kevin-drum/2013/05/groundbreaking-isaac-newton-invention-youve-never-heard/ |access-date=2024-12-21 |website=Mother Jones |language=en-US}}</ref> It was perhaps first expressed formally by [[Roger Cotes]] in 1722. *The combination of different observations taken under the ''same'' conditions contrary to simply trying one's best to observe and record a single observation accurately. The approach was known as the method of averages. This approach was notably used by Newton while studying equinoxes in 1700, also writing down the first of the 'normal equations' known from [[ordinary least squares]],<ref>{{cite arXiv |eprint=0810.4948 |class=physics.hist-ph |first1=Ari |last1=Belenkiy |first2=Eduardo Vila |last2=Echague |title=Groping Toward Linear Regression Analysis: Newton's Analysis of Hipparchus' Equinox Observations |date=2008}}</ref> [[Tobias Mayer]] while studying the [[libration]]s of the Moon in 1750, and by [[Pierre-Simon Laplace]] in his work in explaining the differences in motion of [[Jupiter]] and [[Saturn]] in 1788. *The combination of different observations taken under {{em|different}} conditions. The method came to be known as the method of ''[[least absolute deviation]]''. It was notably performed by [[Roger Joseph Boscovich]] in his work on the shape of the Earth in 1757 and by [[Pierre-Simon Laplace]] for the same problem in 1789 and 1799. *The development of a criterion that can be evaluated to determine when the solution with the minimum error has been achieved. Laplace tried to specify a mathematical form of the [[probability]] density for the errors and define a method of estimation that minimizes the error of estimation. For this purpose, Laplace used a symmetric two-sided exponential distribution we now call [[Laplace distribution]] to model the error distribution, and used the sum of absolute deviation as error of estimation. He felt these to be the simplest assumptions he could make, and he had hoped to obtain the arithmetic mean as the best estimate. Instead, his estimator was the posterior median. ===The method=== [[File:Bendixen - Carl Friedrich Gauß, 1828.jpg|thumb|upright=0.8|[[Carl Friedrich Gauss]]]] The first clear and concise exposition of the method of least squares was published by [[Adrien-Marie Legendre|Legendre]] in 1805.<ref>{{Citation |first=Adrien-Marie |last=Legendre |title=Nouvelles méthodes pour la détermination des orbites des comètes |trans-title=New Methods for the Determination of the Orbits of Comets |language=fr |publisher=F. Didot |location=Paris |year=1805 |url=https://books.google.com/books?id=FRcOAAAAQAAJ |hdl=2027/nyp.33433069112559 |hdl-access=free }}</ref> The technique is described as an algebraic procedure for fitting linear equations to data and Legendre demonstrates the new method by analyzing the same data as Laplace for the shape of the Earth. Within ten years after Legendre's publication, the method of least squares had been adopted as a standard tool in astronomy and geodesy in [[France]], [[Italy]], and [[Prussia]], which constitutes an extraordinarily rapid acceptance of a scientific technique.<ref name=stigler></ref> In 1809 [[Carl Friedrich Gauss]] published his method of calculating the orbits of celestial bodies. In that work he claimed to have been in possession of the method of least squares since 1795.<ref>{{Cite web |last= |date=2015-11-06 |title=The Discovery of Statistical Regression |url=https://priceonomics.com/the-discovery-of-statistical-regression/ |access-date=2023-04-04 |website=Priceonomics |language=en}}</ref> This naturally led to a priority dispute with Legendre. However, to Gauss's credit, he went beyond Legendre and succeeded in connecting the method of least squares with the principles of probability and to the [[normal distribution]]. He had managed to complete Laplace's program of specifying a mathematical form of the probability density for the observations, depending on a finite number of unknown parameters, and define a method of estimation that minimizes the error of estimation. Gauss showed that the [[arithmetic mean]] is indeed the best estimate of the location parameter by changing both the [[probability density]] and the method of estimation. He then turned the problem around by asking what form the density should have and what method of estimation should be used to get the arithmetic mean as estimate of the location parameter. In this attempt, he invented the normal distribution. An early demonstration of the strength of [[Gauss's method]] came when it was used to predict the future location of the newly discovered asteroid [[Ceres (dwarf planet)|Ceres]]. On 1 January 1801, the Italian astronomer [[Giuseppe Piazzi]] discovered Ceres and was able to track its path for 40 days before it was lost in the glare of the Sun. Based on these data, astronomers desired to determine the location of Ceres after it emerged from behind the Sun without solving [[Kepler's laws of planetary motion|Kepler's complicated nonlinear equations]] of planetary motion. The only predictions that successfully allowed Hungarian astronomer [[Franz Xaver von Zach]] to relocate Ceres were those performed by the 24-year-old Gauss using least-squares analysis. In 1810, after reading Gauss's work, Laplace, after proving the [[central limit theorem]], used it to give a large sample justification for the method of least squares and the normal distribution. In 1822, Gauss was able to state that the least-squares approach to regression analysis is optimal in the sense that in a linear model where the errors have a mean of zero, are uncorrelated, normally distributed, and have equal variances, the best linear unbiased estimator of the coefficients is the least-squares estimator. An extended version of this result is known as the [[Gauss–Markov theorem]]. The idea of least-squares analysis was also independently formulated by the American [[Robert Adrain]] in 1808. In the next two centuries workers in the theory of errors and in statistics found many different ways of implementing least squares.<ref>{{cite journal |doi=10.1111/j.1751-5823.1998.tb00406.x |first=J. |last=Aldrich |year=1998|title=Doing Least Squares: Perspectives from Gauss and Yule |journal=International Statistical Review |volume=66 |issue=1 |pages= 61–81|s2cid=121471194 }}</ref> ==Problem statement== The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of ''n'' points (data pairs) <math>(x_i,y_i)\!</math>, ''i'' = 1, …, ''n'', where <math>x_i\!</math> is an [[independent variable]] and <math>y_i\!</math> is a [[dependent variable]] whose value is found by observation. The model function has the form <math>f(x, \boldsymbol \beta)</math>, where ''m'' adjustable parameters are held in the vector <math>\boldsymbol \beta</math>. The goal is to find the parameter values for the model that "best" fits the data. The fit of a model to a data point is measured by its [[errors and residuals in statistics|residual]], defined as the difference between the observed value of the dependent variable and the value predicted by the model: <math display="block">r_i = y_i - f(x_i, \boldsymbol \beta).</math> [[File:Linear Residual Plot Graph.png|thumb|The residuals are plotted against corresponding <math>x</math> values. The random fluctuations about <math>r_i=0</math> indicate a linear model is appropriate.|251x251px]] The least-squares method finds the optimal parameter values by minimizing the [[sum of squared residuals]], <math>S</math>:<ref name=":0" /> <math display="block">S = \sum_{i=1}^n r_i^2.</math> In the simplest case <math>f(x_i, \boldsymbol \beta)= \beta</math> and the result of the least-squares method is the [[arithmetic mean]] of the input data. An example of a model in two dimensions is that of the straight line. Denoting the y-intercept as <math>\beta_0</math> and the slope as <math>\beta_1</math>, the model function is given by <math>f(x,\boldsymbol \beta)=\beta_0+\beta_1 x</math>. See [[Linear least squares #Example|linear least squares]] for a fully worked out example of this model. A data point may consist of more than one independent variable. For example, when fitting a plane to a set of height measurements, the plane is a function of two independent variables, ''x'' and ''z'', say. In the most general case there may be one or more independent variables and one or more dependent variables at each data point. To the right is a residual plot illustrating random fluctuations about <math>r_i=0</math>, indicating that a linear model<math>(Y_i = \beta_0 + \beta_1 x_i + U_i)</math> is appropriate. <math>U_i</math> is an independent, random variable.<ref name=":0">{{Cite book|title=A modern introduction to probability and statistics: understanding why and how|date=2005|publisher=Springer | others=Dekking, Michel, 1946-|isbn=978-1-85233-896-1|location=London|oclc=262680588}}</ref> [[File:Parabolic Residual Plot Graph.png|thumb|247x247px|The residuals are plotted against the corresponding <math>x</math> values. The parabolic shape of the fluctuations about <math>r_i=0</math> indicates a parabolic model is appropriate.]] If the residual points had some sort of a shape and were not randomly fluctuating, a linear model would not be appropriate. For example, if the residual plot had a parabolic shape as seen to the right, a parabolic model <math>(Y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + U_i)</math> would be appropriate for the data. The residuals for a parabolic model can be calculated via <math>r_i=y_i-\hat{\beta}_0-\hat{\beta}_1 x_i-\hat{\beta}_2 x_i^2</math>.<ref name=":0" /> <!-- Also, the residuals may be weighted to take into account differences in the reliability of the measurements. <math> S = \sum_{i=1}^n w_i r_i^2 </math>. This is called '''weighted least squares,''' in contrast to '''ordinary least squares''' in which unit weights are used. --> ==Limitations== This regression formulation considers only observational errors in the dependent variable (but the alternative [[total least squares]] regression can account for errors in both variables). There are two rather different contexts with different implications: *Regression for prediction. Here a model is fitted to provide a prediction rule for application in a similar situation to which the data used for fitting apply. Here the dependent variables corresponding to such future application would be subject to the same types of observation error as those in the data used for fitting. It is therefore logically consistent to use the least-squares prediction rule for such data. *Regression for fitting a "true relationship". In standard [[regression analysis]] that leads to fitting by least squares there is an implicit assumption that errors in the [[independent variable]] are zero or strictly controlled so as to be negligible. When errors in the [[independent variable]] are non-negligible, [[Errors-in-variables models|models of measurement error]] can be used; such methods can lead to [[parameter estimation|parameter estimates]], [[hypothesis testing]] and [[confidence interval]]s that take into account the presence of observation errors in the independent variables.<ref>For a good introduction to error-in-variables, please see {{cite book |last=Fuller |first=W. A. |author-link=Wayne Fuller |title=Measurement Error Models |publisher=John Wiley & Sons |year=1987 |isbn=978-0-471-86187-4 }}</ref> An alternative approach is to fit a model by [[total least squares]]; this can be viewed as taking a pragmatic approach to balancing the effects of the different sources of error in formulating an objective function for use in model-fitting. ==Solving the least squares problem== The [[Maxima and minima|minimum]] of the sum of squares is found by setting the [[gradient]] to zero. Since the model contains ''m'' parameters, there are ''m'' gradient equations: <math display="block">\frac{\partial S}{\partial \beta_j}=2\sum_i r_i\frac{\partial r_i}{\partial \beta_j} = 0,\ j=1,\ldots,m,</math> and since <math>r_i=y_i-f(x_i,\boldsymbol \beta)</math>, the gradient equations become <math display="block">-2\sum_i r_i\frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j}=0,\ j=1,\ldots,m.</math> The gradient equations apply to all least squares problems. Each particular problem requires particular expressions for the model and its [[Partial derivative|partial derivatives]].<ref name=":1">{{Cite book|title=Quantifying measurement: the tyranny of numbers|last=Williams, Jeffrey H. (Jeffrey Huw), 1956-|others=Morgan & Claypool Publishers, Institute of Physics (Great Britain)|isbn=978-1-68174-433-9|location=San Rafael [California] (40 Oak Drive, San Rafael, CA, 94903, USA)|oclc=962422324|date = November 2016}}</ref> ===Linear least squares=== {{main|Linear least squares (mathematics)|l1=Linear least squares}} A regression model is a linear one when the model comprises a [[linear combination]] of the parameters, i.e., <math display="block"> f(x, \boldsymbol \beta) = \sum_{j = 1}^m \beta_j \phi_j(x),</math> where the function <math>\phi_j</math> is a function of <math> x </math>.<ref name=":1" /> Letting <math> X_{ij}= \phi_j(x_{i})</math> and putting the independent and dependent variables in matrices <math> X</math> and <math> Y,</math> respectively, we can compute the least squares in the following way. Note that <math> D</math> is the set of all data.<ref name=":1" /><ref name=":2">{{Cite book|last1=Rencher|first1=Alvin C.|url=https://books.google.com/books?id=0g-PAuKub3QC&pg=PA19|title=Methods of Multivariate Analysis|last2=Christensen|first2=William F.|date=2012-08-15|publisher=John Wiley & Sons|isbn=978-1-118-39167-9|pages=155|language=en}}</ref> <math display="block">L(D, \boldsymbol{\beta}) = \left\|Y - X\boldsymbol{\beta} \right\|^2 = (Y - X\boldsymbol{\beta})^\mathsf{T} (Y - X\boldsymbol{\beta})</math> <math display="block">= Y^\mathsf{T}Y- 2Y^\mathsf{T}X\boldsymbol{\beta} + \boldsymbol{\beta}^\mathsf{T}X^\mathsf{T}X\boldsymbol{\beta} </math> The gradient of the loss is: <math display="block">\frac{\partial L(D, \boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = \frac{\partial \left(Y^\mathsf{T}Y- 2Y^\mathsf{T}X\boldsymbol{\beta} + \boldsymbol{\beta}^\mathsf{T}X^\mathsf{T}X\boldsymbol{\beta}\right)}{\partial \boldsymbol{\beta}} = -2X^\mathsf{T}Y + 2X^\mathsf{T}X\boldsymbol{\beta}</math> Setting the gradient of the loss to zero and solving for <math>\boldsymbol{\beta}</math>, we get:<ref name=":2" /><ref name=":1" /> <math display="block">-2X^\mathsf{T}Y + 2X^\mathsf{T}X\boldsymbol{\beta} = 0 \Rightarrow X^\mathsf{T}Y = X^\mathsf{T}X\boldsymbol{\beta}</math> <math display="block">\boldsymbol{\hat{\beta}} = \left(X^\mathsf{T}X\right)^{-1} X^\mathsf{T}Y</math> ===Non-linear least squares=== {{main|Non-linear least squares}} There is, in some cases, a [[closed-form solution]] to a non-linear least squares problem – but in general there is not. In the case of no closed-form solution, numerical algorithms are used to find the value of the parameters <math>\beta</math> that minimizes the objective. Most algorithms involve choosing initial values for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation: <math display="block">{\beta_j}^{k+1} = {\beta_j}^k+\Delta \beta_j,</math> where a superscript ''k'' is an iteration number, and the vector of increments <math>\Delta \beta_j</math> is called the shift vector. In some commonly used algorithms, at each iteration the model may be linearized by approximation to a first-order [[Taylor series]] expansion about <math> \boldsymbol \beta^k</math>: <math display="block">\begin{align} f(x_i,\boldsymbol \beta) &= f^k(x_i,\boldsymbol \beta) +\sum_j \frac{\partial f(x_i,\boldsymbol \beta)}{\partial \beta_j} \left(\beta_j-{\beta_j}^k \right) \\[1ex] &= f^k(x_i,\boldsymbol \beta) +\sum_j J_{ij} \,\Delta\beta_j. \end{align}</math> The [[Jacobian matrix and determinant|Jacobian]] '''J''' is a function of constants, the independent variable ''and'' the parameters, so it changes from one iteration to the next. The residuals are given by <math display="block">r_i = y_i - f^k(x_i, \boldsymbol \beta)- \sum_{k=1}^{m} J_{ik}\,\Delta\beta_k = \Delta y_i- \sum_{j=1}^m J_{ij}\,\Delta\beta_j.</math> To minimize the sum of squares of <math>r_i</math>, the gradient equation is set to zero and solved for <math> \Delta \beta_j</math>: <math display="block">-2\sum_{i=1}^n J_{ij} \left( \Delta y_i-\sum_{k=1}^m J_{ik} \, \Delta \beta_k \right) = 0,</math> which, on rearrangement, become ''m'' simultaneous linear equations, the '''normal equations''': <math display="block">\sum_{i=1}^{n}\sum_{k=1}^m J_{ij} J_{ik} \, \Delta \beta_k=\sum_{i=1}^n J_{ij} \, \Delta y_i \qquad (j=1,\ldots,m).</math> The normal equations are written in matrix notation as <math display="block">\left(\mathbf{J}^\mathsf{T} \mathbf{J}\right) \Delta \boldsymbol \beta = \mathbf{J}^\mathsf{T}\Delta \mathbf{y}.</math> <!-- or <math display="block">\mathbf{\left(J^TWJ\right) \, \Delta \boldsymbol \beta=J^TW \, \Delta y}</math> if weights are used. --> These are the defining equations of the [[Gauss–Newton algorithm]]. ===Differences between linear and nonlinear least squares=== * The model function, ''f'', in LLSQ (linear least squares) is a linear combination of parameters of the form <math>f = X_{i1}\beta_1 + X_{i2}\beta_2 +\cdots</math> The model may represent a straight line, a parabola or any other linear combination of functions. In NLLSQ (nonlinear least squares) the parameters appear as functions, such as <math>\beta^2, e^{\beta x}</math> and so forth. If the derivatives <math>\partial f / \partial \beta_j</math> are either constant or depend only on the values of the independent variable, the model is linear in the parameters. Otherwise, the model is nonlinear. *Need initial values for the parameters to find the solution to a NLLSQ problem; LLSQ does not require them. *Solution algorithms for NLLSQ often require that the Jacobian can be calculated similar to LLSQ. Analytical expressions for the partial derivatives can be complicated. If analytical expressions are impossible to obtain either the partial derivatives must be calculated by numerical approximation or an estimate must be made of the Jacobian, often via [[finite differences]]. *Non-convergence (failure of the algorithm to find a minimum) is a common phenomenon in NLLSQ. *LLSQ is globally concave so non-convergence is not an issue. *Solving NLLSQ is usually an iterative process which has to be terminated when a convergence criterion is satisfied. LLSQ solutions can be computed using direct methods, although problems with large numbers of parameters are typically solved with iterative methods, such as the [[Gauss–Seidel]] method. *In LLSQ the solution is unique, but in NLLSQ there may be multiple minima in the sum of squares. *Under the condition that the errors are uncorrelated with the predictor variables, LLSQ yields unbiased estimates, but even under that condition NLLSQ estimates are generally biased. These differences must be considered whenever the solution to a nonlinear least squares problem is being sought.<ref name=":1" /> ==Example== Consider a simple example drawn from physics. A spring should obey [[Hooke's law]] which states that the extension of a spring {{mvar|y}} is proportional to the force, ''F'', applied to it. <math display="block">y = f(F,k) = k F</math> constitutes the model, where ''F'' is the independent variable. In order to estimate the [[force constant]], ''k'', we conduct a series of ''n'' measurements with different forces to produce a set of data, <math>(F_i, y_i),\ i=1,\dots,n\!</math>, where ''y<sub>i</sub>'' is a measured spring extension.<ref>{{Cite book | title = Mechanics of Materials | last1 = Gere | first1 = James M. | date = 2013 | publisher = Cengage Learning | last2 = Goodno | first2 = Barry J. | isbn = 978-1-111-57773-5 |edition = 8th | location = Stamford, Conn. | oclc = 741541348}}</ref> Each experimental observation will contain some error, <math>\varepsilon</math>, and so we may specify an empirical model for our observations, <math display="block"> y_i = kF_i + \varepsilon_i. </math> There are many methods we might use to estimate the unknown parameter ''k''. Since the ''n'' equations in the ''m'' variables in our data comprise an [[overdetermined system]] with one unknown and ''n'' equations, we estimate ''k'' using least squares. The sum of squares to be minimized is<ref name=":1" /> <math display="block"> S = \sum_{i=1}^n \left(y_i - k F_i\right)^2. </math> The least squares estimate of the force constant, ''k'', is given by <math display="block">\hat k = \frac{\sum_i F_i y_i}{\sum_i F_i^2}.</math> We assume that applying force '''''causes''''' the spring to expand. After having derived the force constant by least squares fitting, we predict the extension from Hooke's law. ==Uncertainty quantification== In a least squares calculation with unit weights, or in linear regression, the variance on the ''j''th parameter, denoted <math>\operatorname{var}(\hat{\beta}_j)</math>, is usually estimated with <math display="block">\operatorname{var}(\hat{\beta}_j)= \sigma^2\left(\left[X^\mathsf{T}X\right]^{-1}\right)_{jj} \approx \hat{\sigma}^2 C_{jj},</math> <math display="block">\hat{\sigma}^2 \approx \frac S {n-m} </math> <math display="block">C = \left(X^\mathsf{T}X\right)^{-1},</math> where the true error variance ''σ''<sup>2</sup> is replaced by an estimate, the [[reduced chi-squared statistic]], based on the minimized value of the [[residual sum of squares]] (objective function), ''S''. The denominator, ''n'' − ''m'', is the [[Degrees of freedom (statistics)|statistical degrees of freedom]]; see [[Degrees of freedom (statistics)#Effective degrees of freedom|effective degrees of freedom]] for generalizations.<ref name=":1" /> ''C'' is the [[covariance matrix]]. ==Statistical testing== If the [[probability distribution]] of the parameters is known or an asymptotic approximation is made, [[confidence limits]] can be found. Similarly, statistical tests on the residuals can be conducted if the probability distribution of the residuals is known or assumed. We can derive the probability distribution of any linear combination of the dependent variables if the probability distribution of experimental errors is known or assumed. Inferring is easy when assuming that the errors follow a normal distribution, consequently implying that the parameter estimates and residuals will also be normally distributed conditional on the values of the independent variables.<ref name=":1" /> It is necessary to make assumptions about the nature of the experimental errors to test the results statistically. A common assumption is that the errors belong to a normal distribution. The [[central limit theorem]] supports the idea that this is a good approximation in many cases. * The [[Gauss–Markov theorem]]. In a linear model in which the errors have [[expected value|expectation]] zero conditional on the independent variables, are [[uncorrelated]] and have equal [[variance]]s, the best linear [[unbiased]] estimator of any linear combination of the observations, is its least-squares estimator. "Best" means that the least squares estimators of the parameters have minimum variance. The assumption of equal variance is valid when the errors all belong to the same distribution.<ref>{{cite encyclopedia |last1=Hallin |first1=Marc |title=Gauss-Markov Theorem |url=https://onlinelibrary.wiley.com/doi/10.1002/9780470057339.vnn102 |publisher=Wiley |date=2012 |encyclopedia=Encyclopedia of Environmetrics |doi=10.1002/9780470057339.vnn102 |isbn=978-0-471-89997-6 |access-date=18 October 2023}}</ref> *If the errors belong to a normal distribution, the least-squares estimators are also the [[maximum likelihood estimator]]s in a linear model. However, suppose the errors are not normally distributed. In that case, a [[central limit theorem]] often nonetheless implies that the parameter estimates will be approximately normally distributed so long as the sample is reasonably large. For this reason, given the important property that the error mean is independent of the independent variables, the distribution of the error term is not an important issue in regression analysis. Specifically, it is not typically important whether the error term follows a normal distribution. ==Weighted least squares== [[File:Heteroscedasticity Fanning Out .png|thumb|244x244px|"Fanning Out" Effect of Heteroscedasticity]] {{main|Weighted least squares}} A special case of [[generalized least squares]] called '''weighted least squares''' occurs when all the off-diagonal entries of Ω (the correlation matrix of the residuals) are null; the [[variance]]s of the observations (along the covariance matrix diagonal) may still be unequal ([[heteroscedasticity]]). In simpler terms, [[heteroscedasticity]] is when the variance of <math>Y_i</math> depends on the value of <math>x_i</math> which causes the residual plot to create a "fanning out" effect towards larger <math>Y_i</math> values as seen in the residual plot to the right. On the other hand, [[homoscedasticity]] is assuming that the variance of <math>Y_i</math> and variance of <math>U_i</math> are equal.<ref name=":0" /> ==Relationship to principal components== The first [[Principal component analysis|principal component]] about the mean of a set of points can be represented by that line which most closely approaches the data points (as measured by squared distance of closest approach, i.e. perpendicular to the line). In contrast, linear least squares tries to minimize the distance in the <math>y</math> direction only. Thus, although the two use a similar error metric, linear least squares is a method that treats one dimension of the data preferentially, while PCA treats all dimensions equally. ==Relationship to measure theory== Notable statistician [[Sara van de Geer]] used [[empirical process theory]] and the [[Vapnik–Chervonenkis dimension]] to prove a least-squares estimator can be interpreted as a [[Measure (mathematics)|measure]] on the space of [[square-integrable functions]].<ref name=bigcrush>{{cite journal|author-last=van de Geer|author-first=Sara|author-link=Sara van de Geer|title=A New Approach to Least-Squares Estimation, with Applications | journal=[[Annals of Statistics]]| volume=15 | issue=2 | pages=587–602 | date=June 1987 |doi=10.1214/aos/1176350362|s2cid=123088844 |doi-access=free}}</ref> ==Regularization== {{Main|Regularized least squares}} ===Tikhonov regularization=== {{Main|Tikhonov regularization}} In some contexts, a [[Regularization (machine learning)|regularized]] version of the least squares solution may be preferable. [[Tikhonov regularization]] (or [[ridge regression]]) adds a constraint that <math>\left\|\beta\right\|_2^2</math>, the squared [[L2-norm|<math>\ell_2</math>-norm]] of the parameter vector, is not greater than a given value to the least squares formulation, leading to a constrained minimization problem. This is equivalent to the unconstrained minimization problem where the objective function is the residual sum of squares plus a penalty term <math>\alpha \left\|\beta\right\|_2^2</math> and <math>\alpha</math> is a tuning parameter (this is the [[Lagrange multipliers|Lagrangian]] form of the constrained minimization problem).<ref>{{cite arXiv |last=van Wieringen |first=Wessel N. |year=2021 |title=Lecture notes on ridge regression |class=stat.ME |eprint=1509.09169 }}</ref> In a [[Bayesian statistics|Bayesian]] context, this is equivalent to placing a zero-mean normally distributed [[prior distribution|prior]] on the parameter vector. ===Lasso method=== An alternative [[Regularization (machine learning)|regularized]] version of least squares is [[Lasso (statistics)|Lasso]] (least absolute shrinkage and selection operator), which uses the constraint that <math>\|\beta\|_1</math>, the [[L1-norm|L<sub>1</sub>-norm]] of the parameter vector, is no greater than a given value.<ref name=tibsh>{{cite journal |last=Tibshirani |first=R. |author-link = Rob Tibshirani | year=1996 |title=Regression shrinkage and selection via the lasso |journal=Journal of the Royal Statistical Society, Series B |volume=58|issue=1 |pages=267–288 |doi=10.1111/j.2517-6161.1996.tb02080.x |jstor=2346178}}</ref><ref name="ElementsStatLearn">{{cite book |url=http://www-stat.stanford.edu/~tibs/ElemStatLearn/ |title=The Elements of Statistical Learning |last1=Hastie |first1=Trevor |last2=Tibshirani |first2=Robert |last3=Friedman |first3=Jerome H. |author-link1=Trevor Hastie |author-link3=Jerome H. Friedman |edition=second |date=2009 |publisher=Springer-Verlag |isbn=978-0-387-84858-7 |url-status=dead |archive-url=https://web.archive.org/web/20091110212529/http://www-stat.stanford.edu/~tibs/ElemStatLearn/ |archive-date=2009-11-10 }}</ref><ref>{{cite book|last1=Bühlmann|first1=Peter|last2=van de Geer|first2=Sara|author2-link= Sara van de Geer |title=Statistics for High-Dimensional Data: Methods, Theory and Applications|date=2011|publisher=Springer|isbn=9783642201929}}</ref> (One can show like above using Lagrange multipliers that this is equivalent to an unconstrained minimization of the least-squares penalty with <math>\alpha\|\beta\|_1</math> added.) In a [[Bayesian statistics|Bayesian]] context, this is equivalent to placing a zero-mean [[Laplace distribution|Laplace]] [[prior distribution]] on the parameter vector.<ref>{{cite journal|last1=Park|first1=Trevor|last2=Casella|first2=George|author2-link=George Casella| title=The Bayesian Lasso|journal=Journal of the American Statistical Association|date=2008|volume=103|issue=482|pages=681–686|doi=10.1198/016214508000000337|s2cid=11797924}}</ref> The optimization problem may be solved using [[quadratic programming]] or more general [[convex optimization]] methods, as well as by specific algorithms such as the [[least angle regression]] algorithm. One of the prime differences between Lasso and ridge regression is that in ridge regression, as the penalty is increased, all parameters are reduced while still remaining non-zero, while in Lasso, increasing the penalty will cause more and more of the parameters to be driven to zero. This is an advantage of Lasso over ridge regression, as driving parameters to zero deselects the features from the regression. Thus, Lasso automatically selects more relevant features and discards the others, whereas Ridge regression never fully discards any features. Some [[feature selection]] techniques are developed based on the LASSO including Bolasso which bootstraps samples,<ref name=Bolasso>{{cite book|last1=Bach|first1=Francis R|title=Proceedings of the 25th international conference on Machine learning - ICML '08 |chapter=Bolasso |date=2008|pages=33–40|doi=10.1145/1390156.1390161|chapter-url=http://dl.acm.org/citation.cfm?id=1390161|isbn=9781605582054|bibcode=2008arXiv0804.1302B|arxiv=0804.1302|s2cid=609778}}</ref> and FeaLect which analyzes the regression coefficients corresponding to different values of <math>\alpha</math> to score all the features.<ref name=FeaLect>{{cite journal|last1=Zare|first1=Habil|title=Scoring relevancy of features based on combinatorial analysis of Lasso with application to lymphoma diagnosis|journal=BMC Genomics|date=2013|volume=14|issue=Suppl 1 |pages=S14|doi=10.1186/1471-2164-14-S1-S14|pmid=23369194|pmc=3549810 |doi-access=free }}</ref> The ''L''<sup>1</sup>-regularized formulation is useful in some contexts due to its tendency to prefer solutions where more parameters are zero, which gives solutions that depend on fewer variables.<ref name=tibsh/> For this reason, the Lasso and its variants are fundamental to the field of [[compressed sensing]]. An extension of this approach is [[elastic net regularization]]. ==See also== {{Div col|colwidth=25em}} * [[Least-squares adjustment]] * [[Minimum mean square error|Bayesian MMSE estimator]] * [[Gauss–Markov theorem|Best linear unbiased estimator]] (BLUE) * [[Best linear unbiased prediction]] (BLUP) * [[Gauss–Markov theorem]] * [[L2 norm|''L''<sup>2</sup> norm]] * [[Least absolute deviations]] * [[Least-squares spectral analysis]] * [[Measurement uncertainty]] * [[Orthogonal projection]] * [[Proximal gradient methods for learning]] * [[Quadratic loss function]] * [[Root mean square]] * [[Squared deviations from the mean]] {{div col end}} ==References== {{Reflist|30em}} ==Further reading== * {{cite book |first=Å. |last=Björck |isbn=978-0-89871-360-2 |title=Numerical Methods for Least Squares Problems |publisher=SIAM |year=1996 }} * {{cite book |first1=T. |last1=Kariya |first2=H. |last2=Kurata |title=Generalized Least Squares |location=Hoboken |publisher=Wiley |year=2004 |isbn=978-0-470-86697-9 }} * {{cite book |first=D. G. |last=Luenberger |author-link=David Luenberger |chapter=Least-Squares Estimation |title=Optimization by Vector Space Methods |location=New York |publisher=John Wiley & Sons |year=1997 |orig-year=1969 |isbn=978-0-471-18117-0 |pages=78–102 |chapter-url=https://books.google.com/books?id=lZU0CAH4RccC&pg=PA78 }} * {{cite book |first1=C. R. |last1=Rao |author-link=C. R. Rao |first2=H. |last2=Toutenburg |author-link2=Helge Toutenburg |title=Linear Models: Least Squares and Alternatives |series=Springer Series in Statistics |location=Berlin |publisher=Springer |edition=3rd |year=2008 |isbn=978-3-540-74226-5 |url=https://books.google.com/books?id=3LK9JoGEyN4C |display-authors=etal}} * {{cite web |url=https://www.researchgate.net/publication/350838636 |title=Multidirectional regression analysis |last=Van de moortel |first=Koen |date=April 2021 |website= |publisher= |access-date= |quote=}} * {{cite book |first=J. |last=Wolberg |title=Data Analysis Using the Method of Least Squares: Extracting the Most Information from Experiments |location=Berlin |publisher=Springer |year=2005|isbn=978-3-540-25674-8}} ==External links== *{{Commonscat-inline}} {{Least Squares and Regression Analysis|state=expanded}} {{Statistics|correlation|state=collapsed}} {{Authority control}} {{DEFAULTSORT:Least Squares}} [[Category:Least squares| ]] [[Category:Single-equation methods (econometrics)]] [[Category:Optimization algorithms and methods]] [[gl:Mínimos cadrados lineais]] [[vi:Bình phương tối thiểu tuyến tính]]
Edit summary
(Briefly describe your changes)
By publishing changes, you agree to the
Terms of Use
, and you irrevocably agree to release your contribution under the
CC BY-SA 4.0 License
and the
GFDL
. You agree that a hyperlink or URL is sufficient attribution under the Creative Commons license.
Cancel
Editing help
(opens in new window)
Pages transcluded onto the current version of this page
(
help
)
:
Template:Authority control
(
edit
)
Template:Citation
(
edit
)
Template:Cite arXiv
(
edit
)
Template:Cite book
(
edit
)
Template:Cite encyclopedia
(
edit
)
Template:Cite journal
(
edit
)
Template:Cite web
(
edit
)
Template:Commonscat-inline
(
edit
)
Template:Distinguish-redirect
(
edit
)
Template:Div col
(
edit
)
Template:Div col end
(
edit
)
Template:Em
(
edit
)
Template:Least Squares and Regression Analysis
(
edit
)
Template:Longlead
(
edit
)
Template:Main
(
edit
)
Template:Mvar
(
edit
)
Template:Reflist
(
edit
)
Template:Regression bar
(
edit
)
Template:Short description
(
edit
)
Template:Statistics
(
edit
)