# Incorrect results produced by simple subtraction

To reproduce:

In cell A1 enter the value 65941.152 In cell B1 enter the formula A1-65536 The value displayed in cell B1 will be 405.152000000002

Libre Office 5.3.1.2 Linux 4.9.18 64bit

edit retag close merge delete

I should have asked an actual question rather than just showing how to obtain an incorrect result. Question: How do I obtain a correct result?

( 2017-07-07 22:01:34 +0200 )edit

This is the correct result, in the sense that any calculations with floating-point values using computers are necessarily approximate, the error depends on hardware limitations. You should get familiar with this (see e.g. https://en.wikipedia.org/wiki/Floatin...), and be prepared.

( 2017-07-07 22:32:51 +0200 )edit

( 2017-07-07 23:14:43 +0200 )edit

Sort by » oldest newest most voted

If you do the calculations with Excel, and set its precision to enough decimal digits, you will see the error (happening due to binary rounding), too.

more

(The answer expects everybody to know that "computers calculate with binary numbers". The answer below uses the more correct term "dyadic representation".)

There are two effects concerning non-integer arithmetic involved:

-1- There is no way to get generally closed round-trips by conversion between the dyadic and the decimal representation of numbers with fixed lengths (numbers of digits) for either system. Exact conversion from decimal to dyadic is strictly impossible for most non-integer starting values. Exact conversion from dyadic to decimal is only generally possible (in an impractical sense) if as many decimal digits are allowed as dyadic digits were used (which are in practice 52 for most computer applications).

-2- Cancellation of significant digits by subtraction. See this paragraph in the English wikipedia and the article it is contained in. The effect may manifest itself as a stepping forward of insignificant digits.

Concerning your example this was the final effect.
You may also try =65941.008- 65941.0016 with 18 or more decimal places displayed.

more

From 5.4 there is a new function ROUNDSIG() Add function to Calc that rounds to significant digits, as the issue usually happens on the last number of the number's precision (15 digit plus sign), a =ROUNDSIG(A1-65536;14) should solve much of the cases, like this one, with only one significant digit it's lost.

more