Editing Numerical differentiation (section)

==Step size==
{{see also|Adaptive step size}}
[[Image:AbsoluteErrorNumericalDifferentiationExample.png|thumb|300px|Example showing the difficulty of choosing {{mvar|h}} due to both rounding error and formula error]]

An important consideration in practice when the function is calculated using [[floating-point arithmetic]] of finite precision is the choice of step size, {{mvar|h}}. If chosen too small, the subtraction will yield a large [[rounding error]]. In fact, all the finite-difference formulae are [[ill-conditioned]]<ref name=Fornberg1>Numerical Differentiation of Analytic Functions, B Fornberg – ACM Transactions on Mathematical Software (TOMS), 1981.</ref> and due to cancellation will produce a value of zero if {{mvar|h}} is small enough.<ref name=SquireTrapp1>Using Complex Variables to Estimate Derivatives of Real Functions, W. Squire, G. Trapp – SIAM REVIEW, 1998.</ref> If too large, the calculation of the slope of the secant line will be more accurately calculated, but the estimate of the slope of the tangent by using the secant could be worse.<ref name="edo2021">{{Cite book|url= http://flowlab.groups.et.byu.net/mdobook.pdf | title=Engineering Design Optimization | last1=Martins|first1=Joaquim R. R. A. | last2=Ning|first2=Andrew | date=2021-10-01 | publisher=Cambridge University Press | isbn=978-1108833417|language=en}}</ref>

For basic central differences, the optimal step is the cube-root of [[machine epsilon]].<ref>Sauer, Timothy (2012). ''Numerical Analysis''. Pearson. p.248.</ref>
For the numerical derivative formula evaluated at {{mvar|x}} and {{math|''x'' + ''h''}}, a choice for {{mvar|h}} that is small without producing a large rounding error is <math>\sqrt{\varepsilon} x</math> (though not when ''x'' = 0), where the [[machine epsilon]] {{mvar|ε}} is typically of the order of {{val|2.2e−16}} for [[double precision]].<ref>Following ''[[Numerical Recipes]] in C'', [http://www.nrbook.com/a/bookcpdf/c5-7.pdf Chapter 5.7].</ref> A formula for {{mvar|h}} that balances the rounding error against the secant error for optimum accuracy is<ref>[http://www.uio.no/studier/emner/matnat/math/MAT-INF1100/h10/kompendiet/kap11.pdf p. 263].</ref>
<math display="block">h = 2\sqrt{\varepsilon\left|\frac{f(x)}{f''(x)}\right|}</math>
(though not when <math>f''(x) = 0</math>), and to employ it will require knowledge of the function.

For computer calculations the problems are exacerbated because, although {{mvar|x}} necessarily holds a [[representable floating-point number]] in some precision (32 or 64-bit, ''etc''.), {{math|''x'' + ''h''}} almost certainly will not be exactly representable in that precision. This means that {{math|''x'' + ''h''}} will be changed (by rounding or truncation) to a nearby machine-representable number, with the consequence that {{math|(''x'' + ''h'') − ''x''}} will ''not'' equal {{mvar|h}}; the two function evaluations will not be exactly {{mvar|h}} apart. In this regard, since most decimal fractions are recurring sequences in binary (just as 1/3 is in decimal) a seemingly round step such as {{math|1=''h'' = 0.1}} will not be a round number in binary; it is 0.000110011001100...<sub>2</sub> A possible approach is as follows:
 h     := sqrt(eps) * x;
 xph   := x + h;
 dx    := xph - x;
 slope := (F(xph) - F(x)) / dx;
However, with computers, [[compiler optimization]] facilities may fail to attend to the details of actual computer arithmetic and instead apply the axioms of mathematics to deduce that {{math|''dx''}} and {{mvar|h}} are the same. With [[C (programming language)|C]] and similar languages, a directive that {{math|''xph''}} is a [[volatile variable]] will prevent this.