To reproduce:
In cell A1 enter the value 65941.152
In cell B1 enter the formula A1-65536
The value displayed in cell B1 will be 405.152000000002
Libre Office 5.3.1.2 Linux 4.9.18 64bit
I should have asked an actual question rather than just showing how to obtain an incorrect result. Question: How do I obtain a correct result?
This is the correct result, in the sense that any calculations with floating-point values using computers are necessarily approximate, the error depends on hardware limitations. You should get familiar with this (see e.g. Floating-point arithmetic - Wikipedia), and be prepared.
The OQer may get some additional hints from the secondary answer I added, in specific from the links placed there. Yet another link.
If you do the calculations with Excel, and set its precision to enough decimal digits, you will see the error (happening due to binary rounding), too.
(The answer expects everybody to know that “computers calculate with binary numbers”. The answer below uses the more correct term “dyadic representation”.)
There are two effects concerning non-integer arithmetic involved:
-1- There is no way to get generally closed round-trips by conversion between the dyadic and the decimal representation of numbers with fixed lengths (numbers of digits) for either system. Exact conversion from decimal to dyadic is strictly impossible for most non-integer starting values. Exact conversion from dyadic to decimal is only generally possible (in an impractical sense) if as many decimal digits are allowed as dyadic digits were used (which are in practice 52 for most computer applications).
-2- Cancellation of significant digits by subtraction. See this paragraph in the English wikipedia and the article it is contained in. The effect may manifest itself as a stepping forward of insignificant digits.
Concerning your example this was the final effect.
You may also try =65941.008- 65941.0016 with 18 or more decimal places displayed.
From 5.4 there is a new function ROUNDSIG() Add function to Calc that rounds to significant digits, as the issue usually happens on the last number of the number’s precision (15 digit plus sign), a =ROUNDSIG(A1-65536;14) should solve much of the cases, like this one, with only one significant digit it’s lost.
be aware that:
- in above sample no digit is ‘lost’, as there never haven’t been more than 8,
- roundsig isn’t clean yet, try e.g. ‘=RAWSUBTRACT(ROUNDSIG(0,3;15);0,3)’ and see the result ~5,5511E-17 off,
- rounding at the outer limit between value and fp-artifacts sometimes fails, e.g. ‘=RAWSUBTRACT(ROUNDSIG(SUM(0,1;0,2);16);0,3)’ will show a doubled error,
- e.g. =ROUNDSIG(65941,152-65941;14)’ will fail with 0,15200000000186,
- thus take as rounding parameter maximal 15 reduced by the number of decimal ranges crossed between operands and result,
- also that may fail for results like x,999999999994999,
- there are many other oddyties, as e.g. if you key in ‘=ROUNDSIG(10,99999999999499;13)’ that is shown as ‘=ROUNDSIG(10,999999999995;13)’ but calculated to 10,9999999999900, while keying in
. ‘=ROUNDSIG(10,999999999995;13)’ produces 11,0000000000000,
alltogether ‘a minefield’, and there isn’t yet a known safe path through it