Editing Logistic regression (section)

=== Iteratively reweighted least squares (IRLS) ===

Binary logistic regression (<math>y=0</math> or <math> y=1</math>) can, for example, be calculated using ''iteratively reweighted least squares'' (IRLS), which is equivalent to maximizing the [[log-likelihood]] of a [[Bernoulli distribution|Bernoulli distributed]]  process using [[Newton's method]]. If the problem is written in vector matrix form, with parameters <math>\mathbf{w}^T=[\beta_0,\beta_1,\beta_2, \ldots]</math>, explanatory variables <math>\mathbf{x}(i)=[1, x_1(i), x_2(i), \ldots]^T</math> and expected value of the Bernoulli distribution <math>\mu(i)=\frac{1}{1+e^{-\mathbf{w}^T\mathbf{x}(i)}}</math>, the parameters <math>\mathbf{w}</math> can be found using the following iterative algorithm:

:<math>\mathbf{w}_{k+1} = \left(\mathbf{X}^T\mathbf{S}_k\mathbf{X}\right)^{-1}\mathbf{X}^T \left(\mathbf{S}_k \mathbf{X} \mathbf{w}_k + \mathbf{y} - \mathbf{\boldsymbol\mu}_k\right) </math>

where <math>\mathbf{S}=\operatorname{diag}(\mu(i)(1-\mu(i)))</math> is a diagonal weighting matrix, <math>\boldsymbol\mu=[\mu(1), \mu(2),\ldots]</math> the vector of expected values,

:<math>\mathbf{X}=\begin{bmatrix}
1 & x_1(1) & x_2(1) & \ldots\\
1 & x_1(2) & x_2(2) & \ldots\\
\vdots & \vdots & \vdots   
\end{bmatrix}</math>

The regressor matrix and <math>\mathbf{y}(i)=[y(1),y(2),\ldots]^T</math> the vector of response variables. More details can be found in the literature.<ref>{{cite book|last1=Murphy|first1=Kevin P.|title=Machine Learning – A Probabilistic Perspective|publisher=The MIT Press|date=2012|page=245|isbn=978-0-262-01802-9}}</ref>