Editing Conjugate gradient method (section)

== Advantages and disadvantages ==
The advantages and disadvantages of the conjugate gradient methods are summarized in the lecture notes by Nemirovsky and BenTal.<ref name=":02">{{Cite web |last=Nemirovsky and Ben-Tal |date=2023 |title=Optimization III: Convex Optimization |url=http://www2.isye.gatech.edu/~nemirovs/OPTIIILN2023Spring.pdf}}</ref>{{Rp|location=Sec.7.3}}

=== A pathological example ===

This example is from <ref>{{Cite web |last=Pennington |first=Fabian Pedregosa, Courtney Paquette, Tom Trogdon, Jeffrey |title=Random Matrix Theory and Machine Learning Tutorial |url=https://random-matrix-learning.github.io/ |access-date=2023-12-05 |website=random-matrix-learning.github.io |language=en-ca}}</ref>
Let <math display="inline">t \in (0, 1)</math>, and define<math display="block">W= \begin{bmatrix}
 t & \sqrt{t} & & & & \\
 \sqrt{t} & 1+t & \sqrt{t} & & & \\
 & \sqrt{t} & 1+t & \sqrt{t} & & \\
 & & \sqrt{t} & \ddots & \ddots & \\
 & & & \ddots & & \\
 & & & & & \sqrt{t} \\
 & & & & \sqrt{t} & 1+t
 \end{bmatrix}, \quad b=\begin{bmatrix}
 1 \\
 0 \\
 \vdots \\
 0
 \end{bmatrix}</math>Since <math>W</math> is invertible, there exists a unique solution to <math display="inline">W x = b </math>. Solving it by conjugate gradient descent gives us rather bad convergence:<math display="block">\|b- Wx_k\|^2 = (1/t)^{k}, \quad \|b- Wx_n\|^2 =0</math>In words, during the CG process, the error grows exponentially, until it suddenly becomes zero as the unique solution is found.