45.95 - 46.57 = -0.619999999999997 (should be -0.62 exactly; no rounding)

Frequently asked questions - Calc: Accuracy problem - The Document Foundation Wiki


calc: rounding: general precision problem?
calc: are decimal correct calculations possible?
Unexpected decimal precision error in Calc
etc etc

128312 – Calculation involing some decimals return incorrect floats with 12 d.p.

2 Likes

The decimal point is the cause of the calculation error! So =4595–4657 correctly yields –62.00000000.

Oh your understanding is growing … slowly … binary integers are perfectly suited to substitue decimal integers … substituting decimal fractions by binary fractions is not, many become “endless iterations”, need to be shortened, rounded, often rounded again after operations, and that sometimes fails ( from a decimal POV ). Also pls. look up “evil cancellation” why this more often and more significantly harms subtractions than additions.

There is a little terminological problem.
“Integer” should be understood as a mathematical term. This way there neither are “binary integers” nor “decimal integers”.
In what way we prefer to represent an integer number when writing on a blackboard or when designing an FPU is a matter of tradition or a technical decision.
Currently an ordinary FPU has a maximum of 53 dyadic digits (bits) of which the first one isn’t stored because it isn’t significant (always I) due to normalization.
Anyway the largest integer that can be exactly represented this way is
IbIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII * 2^52, and the question whether this actually is to be interpreted as an integer or as a rounded dyadic fraction with more than 52 digits behind the dyadic delimiter b (regarding the dyadic exponent) can only be answered based on the context.
FPU register content I * 2^99 may or may not represent an integer. Donno.
In short: For a current FPU and for any practical purpose 52 significant bits are enough precision, and relative rounding errors in the range of few*10^-16 can simply be tolerated.

sorry Lupp, I just wanted to keep short because again contenders start to complain about trolling … “binary encoded integers” and “decimal encoded integers”, and surely only in the range both versions are precisely representable. The big difference is to fractions , where binary encoded decimal fractions inherit special properties which make calculations difficult … as we both know …

:slight_smile:

What is tolerable I disagree, one can deal with small deviations, exponential explosions become difficult.

For everyday use an integer is simply a floatingpoint fraction with a sufficiently large order of magnitude. If missing (not in register) digits would all be 0 or if some bits at the end of the queue are only present due to small errors in preceding calculations we cannot know.
Use a CAS if you need one. It may (again limited, of course) be able to perform algebraic and numerical (with fractions) reductions without any numeric error, and deliver an appoximative decimal fraction in a last step if ordered.
If you feel capable of higher level programming, create an interface between Calc and Maxima/gnuplot. :slightly_smiling_face:

heeeyyy!!! don’t trigger me into trolling … you are upsetting your colleagues … if … and it might be it already happened … I’d like to improve a spreadsheet, or … if … and it might be it already happened … I’d like to use a spreadsheet with ( limited ) decimal capabilities … I’d choose Gnumeric in the decimal64 variant. A clear codebase, decimal available with one parameter in compilation, compiles in some seconds where LO Calc AFAIR used 30 minutes … quicker in use for nearly all functionalities except conditional formattings …

You may have noticed that I stopped posting here or contributing to Calc years ago. That was not only because of Mike Kaganski’s unpleasant “tone,” but also because I simply liked Gnumeric better. While 20 developers here spend years tinkering with tiny problems, and asking if they have vanished by chance the one year or thr other there one! maintainer goes into business and integrates decimal datatype within 8 weeks… I am eternally grateful to the guy who suggested somewhere in a bug report to adopt Gnumeric’s math engine. Without him, I might never have found it.
So, as far as I’m concerned, I’m not trolling around here anymore, but if I get too annoyed with mocking comments, there will be answers.
:slight_smile: :wink: