Ask Your Question

Incorrect results produced by simple subtraction

asked 2017-07-07 21:16:32 +0200

CC2099 gravatar image

To reproduce:

In cell A1 enter the value 65941.152 In cell B1 enter the formula A1-65536 The value displayed in cell B1 will be 405.152000000002

Libre Office Linux 4.9.18 64bit

edit retag flag offensive close merge delete


I should have asked an actual question rather than just showing how to obtain an incorrect result. Question: How do I obtain a correct result?

CC2099 gravatar imageCC2099 ( 2017-07-07 22:01:34 +0200 )edit

This is the correct result, in the sense that any calculations with floating-point values using computers are necessarily approximate, the error depends on hardware limitations. You should get familiar with this (see e.g., and be prepared.

Mike Kaganski gravatar imageMike Kaganski ( 2017-07-07 22:32:51 +0200 )edit

The OQer may get some additional hints from the secondary answer I added, in specific from the links placed there. Yet another link.

Lupp gravatar imageLupp ( 2017-07-07 23:14:43 +0200 )edit

3 Answers

Sort by » oldest newest most voted

answered 2017-07-07 21:29:57 +0200

If you do the calculations with Excel, and set its precision to enough decimal digits, you will see the error (happening due to binary rounding), too.

edit flag offensive delete link more

answered 2017-07-07 23:11:47 +0200

Lupp gravatar image

updated 2017-07-08 00:09:28 +0200

(The answer expects everybody to know that "computers calculate with binary numbers". The answer below uses the more correct term "dyadic representation".)

There are two effects concerning non-integer arithmetic involved:

-1- There is no way to get generally closed round-trips by conversion between the dyadic and the decimal representation of numbers with fixed lengths (numbers of digits) for either system. Exact conversion from decimal to dyadic is strictly impossible for most non-integer starting values. Exact conversion from dyadic to decimal is only generally possible (in an impractical sense) if as many decimal digits are allowed as dyadic digits were used (which are in practice 52 for most computer applications).

-2- Cancellation of significant digits by subtraction. See this paragraph in the English wikipedia and the article it is contained in. The effect may manifest itself as a stepping forward of insignificant digits.

Concerning your example this was the final effect.
You may also try =65941.008- 65941.0016 with 18 or more decimal places displayed.

edit flag offensive delete link more

answered 2017-07-08 01:50:04 +0200

m.a.riosv gravatar image

From 5.4 there is a new function ROUNDSIG() Add function to Calc that rounds to significant digits, as the issue usually happens on the last number of the number's precision (15 digit plus sign), a =ROUNDSIG(A1-65536;14) should solve much of the cases, like this one, with only one significant digit it's lost.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-07-07 21:16:32 +0200

Seen: 487 times

Last updated: Jul 08 '17