Editing Round-off error (section)

=== Measuring roundoff error by using machine epsilon ===

The machine epsilon <math>\epsilon_\text{mach}</math> can be used to measure the level of roundoff error when using the two rounding rules above. Below are the formulas and corresponding proof.<ref name="Forrester_2018"/> The first definition of machine epsilon is used here.

==== Theorem ====
# Round-by-chop: <math>\epsilon_\text{mach} = \beta^{1-p}</math>
# Round-to-nearest: <math>\epsilon_\text{mach} = \frac{1}{2}\beta^{1-p}</math>

==== Proof ====
Let <math>x=d_{0}.d_{1}d_{2} \ldots d_{p-1}d_{p} \ldots \times \beta^{n} \in \mathbb{R}</math> where <math>n \in [L, U]</math>, and let <math>fl(x)</math> be the floating-point representation of <math>x</math>.  
Since round-by-chop is being used, it is
<math display="block"> \begin{align}
\frac{|x-fl(x)|}{|x|} &= \frac{|d_{0}.d_{1}d_{2}\ldots d_{p-1}d_{p}d_{p+1}\ldots \times \beta^{n} - d_{0}.d_{1}d_{2}\ldots d_{p-1} \times \beta^{n}|}{|d_{0}.d_{1}d_{2}\ldots \times \beta^{n}|}\\
&= \frac{|d_{p}.d_{p+1} \ldots \times \beta^{n-p}|}{|d_{0}.d_{1}d_{2}\ldots \times \beta^{n}|}\\
&= \frac{|d_{p}.d_{p+1}d_{p+2}\ldots|}{|d_{0}.d_{1}d_{2}\ldots|} \times \beta^{-p}
\end{align}</math>
In order to determine the maximum of this quantity, there is a need to find the maximum of the numerator and the minimum of the denominator. Since <math>d_{0}\neq 0</math> (normalized system), the minimum value of the denominator is <math>1</math>. The numerator is bounded above by <math>(\beta-1).(\beta-1){\overline{(\beta-1)}}=\beta </math>. Thus, <math>\frac{|x-fl(x)|}{|x|} \leq \frac{\beta}{1} \times \beta^{-p} = \beta^{1-p}</math>. Therefore, <math>\epsilon=\beta^{1-p}</math> for round-by-chop.
The proof for round-to-nearest is similar.
* Note that the first definition of machine epsilon is not quite equivalent to the second definition when using the round-to-nearest rule but it is equivalent for round-by-chop.