Editing Gradient descent (section)

=== Geometric behavior and residual orthogonality ===
In steepest descent applied to solving <math> A \vec{x} = \vec{b} </math>, where <math> A </math> is symmetric positive-definite, the residual vectors <math> \vec{r}_k = \vec{b} - A\vec{x}_k </math> are orthogonal across iterations:

:<math>
\vec{r}_{k+1} \cdot \vec{r}_k = 0.
</math>

Because each step is taken in the steepest direction, steepest-descent steps
alternate between directions aligned with the extreme axes of the elongated
level sets.  When <math>\kappa(A)</math> is large, this produces a
characteristic zig-zag path. The poor conditioning of <math> A </math> is the primary cause of the slow convergence, and orthogonality of successive residuals reinforces this alternation.

[[File:Steepest descent convergence path for A = 2 2, 2 3.png|thumb|Convergence path of steepest descent method for A = [[2, 2], [2, 3]]]]

As shown in the image on the right, steepest descent converges slowly due to the high condition number of <math> A </math>, and the orthogonality of residuals forces each new direction to undo the overshoot from the previous step. The result is a path that zigzags toward the solution. This inefficiency is one reason conjugate gradient or preconditioning methods are preferred.<ref>{{Cite book | author1=Holmes, M. | title=Introduction to Scientific Computing and Data Analysis, 2nd Ed | year=2023 | publisher=Springer | isbn=978-3-031-22429-4 }}</ref>