Editing Newton's method in optimization (section)

==Some caveats==

Newton's method, in its original version, has several caveats:

# It does not work if the Hessian is not invertible. This is clear from the very definition of Newton's method, which requires taking the inverse of the Hessian.
# It may not converge at all, but can enter a cycle having more than 1 point. See the {{slink|Newton's method|Failure analysis}}.
# It can converge to a saddle point instead of to a local minimum, see the section "[[#Geometric_interpretation|Geometric interpretation]]" in this article.

The popular modifications of Newton's method, such as quasi-Newton methods or Levenberg-Marquardt algorithm mentioned above, also have caveats:

For example, it is usually required that the cost function is (strongly) convex and the Hessian is globally bounded or Lipschitz continuous, for example this is mentioned in the section "Convergence" in this article. If one looks at the papers by Levenberg and Marquardt in the reference for [[Levenberg–Marquardt algorithm]], which are the original sources for the mentioned method, one can see that there is basically no theoretical analysis in the paper by Levenberg, while the paper by Marquardt only analyses a local situation and does not prove a global convergence result.  One can compare with [[Backtracking line search]] method for Gradient descent, which has good theoretical guarantee under more general assumptions, and can be implemented and works well in practical large scale problems such as Deep Neural Networks.