Editing Ridge regression (section)

==Determination of the Tikhonov factor==
The optimal regularization parameter <math>\alpha</math> is usually unknown and often in practical problems is determined by an ''ad hoc'' method. A possible approach relies on the Bayesian interpretation described below. Other approaches include the [[discrepancy principle]], [[cross-validation (statistics)|cross-validation]], [[L-curve method]],<ref>P. C. Hansen, "The L-curve and its use in the
numerical treatment of inverse problems", [https://www.sintef.no/globalassets/project/evitameeting/2005/lcurve.pdf]</ref> [[restricted maximum likelihood]] and [[unbiased predictive risk estimator]]. [[Grace Wahba]] proved that the optimal parameter, in the sense of [[cross-validation (statistics)#Leave-one-out cross-validation|leave-one-out cross-validation]] minimizes<ref>{{cite journal |last=Wahba |first=G. |year=1990 |title=Spline Models for Observational Data |journal=CBMS-NSF Regional Conference Series in Applied Mathematics |publisher=Society for Industrial and Applied Mathematics |bibcode=1990smod.conf.....W }}</ref><ref>{{cite journal |last3=Wahba |first3=G. |first1=G. |last1=Golub |first2=M. |last2=Heath |year=1979 |title=Generalized cross-validation as a method for choosing a good ridge parameter |journal=Technometrics |volume=21 |issue=2 |pages=215–223 |url=http://www.stat.wisc.edu/~wahba/ftp1/oldie/golub.heath.wahba.pdf |doi=10.1080/00401706.1979.10489751}}</ref>
<math display="block">G = \frac{\operatorname{RSS}}{\tau^2}
= \frac{\left\|X \hat{\beta} - y\right\|^2}{ \left[\operatorname{Tr}\left(I - X\left(X^\mathsf{T} X + \alpha^2 I\right)^{-1} X^\mathsf{T}\right)\right]^2},</math>
where <math>\operatorname{RSS}</math> is the [[residual sum of squares]], and <math>\tau</math> is the [[effective number of degrees of freedom]].

Using the previous SVD decomposition, we can simplify the above expression:
<math display="block">\operatorname{RSS} = \left\| y - \sum_{i=1}^q (u_i' b) u_i \right\|^2 + \left\| \sum _{i=1}^q \frac{\alpha^2}{\sigma_i^2 + \alpha^2} (u_i' b) u_i \right\|^2,</math>
<math display="block">\operatorname{RSS} = \operatorname{RSS}_0 + \left\| \sum_{i=1}^q \frac{\alpha^2}{\sigma_i^2 + \alpha^2} (u_i' b) u_i \right\|^2,</math>
and
<math display="block">\tau = m - \sum_{i=1}^q \frac{\sigma_i^2}{\sigma_i^2 + \alpha^2}
= m - q + \sum_{i=1}^q \frac{\alpha^2}{\sigma _i^2 + \alpha^2}.</math>