calc: are decimal correct calculations possible?

@ajlittoz: ‘In a way, you’re defining an algorithm over the set.’ - yes, may be that describes my understanding and approach, IEEE doubles are a subset of values, and do have a math working inside them, except for a few spinoffs at underflow or overflow, NaN’s and so on, and at cancellation, of which cancellation causes most irritations for users,
if one now takes from this set the ‘subset’ which corresponds with ‘closest representation’ to decimal numbers with 15 significant digits, one can define a mathematics over it which computes somewhat less exactly with irrational values, but within its definition range by rounding artifacts from representation inaccuracies and operations and !cancellation can largely be eliminated, (limited to a ULP of the result!?),

this i would see as the right step to make calc a tool for users to solve everyday tasks instead of a binary adventure guessing and researching why - from the user’s point of view - obviously wrong results are called correct,

@mikekaganski: would you consider re-testing your ‘Pythagoras-pi()’ sample with factors of 3*pi() in col. B, and then re-think if this special case of a calculation with an irrational value holding ‘by chance!’ is a justifying argument to block out attempts to make all! calculations with qualified 16 digit decimal values correct?

@newbie-02: would you ever consider re-thinking if it is reasonable to stop spreading lies? Your comment claims that I am blocking something. In fact, I asked you to send your changes to review in gerrit, multiple times. The net result is just blahblahblah, and silly questions like “I’m doing something in source files, I don’t show you but describe in words as if you could get something useful out of it, tell me what specifically should I change”, which I decided to ignore until you follow the advise to use gerrit to discuss stuff.

i’m in the process to review and stabilize my changes and sort out garbage, would like to present it in a proper working state to avoid such simple counterarguments as your ‘Pythagoras-pi()’ problem. in theese steps i found it failing with my patches, but failing in standard calc too (with a factor of 3*pi() ). thus if you insist your sample should hold it’s useless - at this state - to look at my work. if you re-consider your sample being allowed to catch small harming from something which improves other results it makes sense to proceed.
‘spreading lies’? - didn’t do,
‘blocking something’? - yes, you blocked my work with the ‘in sheet rounding’ and declared it nonsense based on the Pythagoras-pi() sample, pretending calc would hold for scientific calculations with irrational numbers - which is not true ‘in general’, and ignoring that standard calc already uses massive result rounding in plenty cases.

yes, you blocked my work with the ‘in sheet rounding’

You just did that again. You have asked for discussion, and I provided you with an argument. If you meant “please see how awesome I am, everyone”, and didn’t expect disagreements in answers, then it’s funny. Otherwise, any counter-argument is not a blocker, but just some data to think about, and possibly fix.

Posting unfinished stuff to gerrit is the only way to discuss code changes, e.g. asking how to best fix some code problem. It doesn’t need to be finished.

hello @mikekaganski,
if / when you answer to / about ‘the matter’ you are usually very good, when talking about the ‘how to do’ you are often too rough against others and too mimosic about yourself, and we start endless debates.
to avoid that: asking only ‘for the matter’: could a improvement be acceptable if it fails with the ‘Pythagoras-pi()’ problem as calc in standard doe’s similar just with different factors, or is it a blocker?

I suppose that it would be up to @erAck to decide, not me.

I’m reluctant to get involved in this pointless discussion!

As an engineer, I’ve long since categorized mathematical principles as belonging to the ideological ghetto (ideology) that cannot even begin to explain physical, i.e., natural events. With endless effort, attempts are made nonetheless, which are bound to fail. When approaching technology, one must employ the highest levels of mathematics: defining tolerance ranges, maintaining unknown variables, describing probabilities, investigating deviations, evaluating cluster formations as well as uniqueness. In contrast, mathematics is an attempt to simplify something that is naturally and technically and physically complex. For example, a circle is an extreme view of an ellipse at a single, infinitely precise angle. This cannot exist due to external influences and internal changes. The circle, the square, the cylinder, etc., are idealized, unnatural figures. Numbers, too, are merely an auxiliary construct to effect something that lies on a mentally delimited straight line (rules, idealisms, theories). In doing so, one leaves our real world and wanders through constructed worlds. It is a mirror labyrinth, the entrance clearly visible, the exits imagined in advance, and the paths obscured. Getting lost is tolerated. Accepting help — and seduction — is accepted. The technician narrows down an expected result, defines the tolerated range in advance, and looks for outliers and plausible cloud formations in a host of numerous tests, measurements, and experiments. This is rarely learned, since this approach comprises everything from theoretical foundations to advanced experiences and the communications of others. And it involves many mishaps. Strictly speaking, this approach exceeds the limited capabilities of any human brain, which is why any ideology easily takes root there. Like any mathematical construct/mistruct 🫣

hello @koyotak,
nice idea, however out of track - from my POV. We have math, and it’s conclusive in itself. We use math to compute / predict physics and the real world, it works quite well as long as we accounr our imprecise knowledge, our imprecise measurements.
Math requires “unlimited”, in particular unlimited fractions, in the steps to limited sizes of paper, limited denominators in decimal math, limited digits on a desk calculator … we loose the qualification for arithmetic, and need to learn to live with imprecisions / rounding in the system of math itself. It - often - works while staying with the decimal system we have been trained to in school, it becomes difficult once we try to substitute decimal fractions with approximated binary fractions, a different rounding principle … here it’s overstretching the brain of quite some people which then like to become arrogant with “read Goldberg” … whom they themself don’t understand …
Here is the main issue for calc … and other spreadsheets … they opted for binary datatypes reg. “speed speed speed” ( in a time when speed was relevant ), and neglect A.) the aftermath and harm in discussions like this, and B.) alternative datatypes which are more “human friendly”, and C.) that these datatypes are quicker in the costly operations of input / output from / to decimal strings … This can happen to people who understand that 0.2 + 0.1 != 0.3, but don’t understand why, and why this is irritating to other people, who nevertheless try to gloss over it and are only partially successful in doing so…
Building a spreadsheet can work, once we have a clear concept. Mimic Excel is clear expressed, however unclear as nobody knows about the concepts of excel. Introducing a new math, and partly covering it towards decimal doesn’t work out as we see since decades, mimic fractions could work to good degree, however we don’t have unlimited space and most people are overtaxed with them, mimicing “decimal” could work out and match peoples expectations … however calc devlopers aren’t - yet - ready or skilled to do so …

Just use a software aimed to do math:

Calc uses, what the typical processors (binary IEEE) representation of numbers supports.
.
But I don’t think it is useful to extend this long thread after 4 years, as nothing new happened (and I promise: will not happen in the next 5 years.)

“not extend” - agree, think someone else started,
“nothing will happen” - however I’m sure to continuedly be asked every day or the other to check if the reported bugs have vanished by their own …

Don’t tell me, tell others, a warning banner at start of Calc:
“There is nothing exact in floating-point calculations in Calc, …” - Mike Kaganski in “comment #5 in Calc Round Down is Rounding Up for some values”, documentfoundation.org, 2023. [Online]. Available: 154792 – Calc Round Down is Rounding Up for some values, retr: 2025-05-13
And tell them that they are qualified for single calculations, not to serve as spreadsheets ( I didn’t check all of them ).

A spreadsheet program may be used for a variety of applications unforeseeable from the developers. Therefore the specification must be generic since it is impossible to predict the numerical range of the calculations. Introducing a limitation would severely impact the usefulness of the program. Therefore to widen the range, floating-point is used.

One of the frequent applications is business: accounting, invoices, asset tracking, … This application is fundamentally based on integer arithmetic. This is quite antagonist with the use of floating-point.

Yes, accounting is integer, even if you see what seems to be decimals. The smallest “quantum” is a cent (or other minimal currency subunit) and all amounts are multiple of this quantum. Only integer arithmetic can guarantee no rounding is introduced and lead to the “least astonishment” of the customer (for instance the sum of raw price and VAT should be equal to the final price even in the last decimal).

Consequently, all your calculations should be done in integer mode.

This is possible over IEEE-754 within a certain range because the 80-bit variant offers 64 bits integer part. Some precautions must be provisioned to avoid “integer overflow”:

  • addition and subtraction are quite safe as numbers in the higher magnitude end of the interval are quite rare

  • multiplication leaves you 32 bits before entering the “floating-point domain”; remember you must scale the result because is a now a multiple of a cent of a cent (i.e. you must drop 2 least significant decimal digits and perhaps introduce a rounding there to revert to your conventional unit, aka. divide by 100)

    This may be insufficient if you’re tracking a country budget to the cent or big civil engineering projects. In this case, you need to change your basic unit and accept some inaccuracy in the descriptive power of your sheet.

  • division is the most difficult operation because the hardware will nearly systematically give you a floating point result; you must first convert to an integer with CEILING(), FLOOR(), INT(), ROUND(), ROUNDDOWN() or ROUNDUP() and do same scaling

Details of computations are described extensively in The Art of Computer Programming, Vol. 2, Seminumerical Algorithms (by Donald Knuth).

As said, all calculations are done in integer mode. Display (fixed point with 2 decimals) and internal representation (integer) are separate, though related, things. Formatting will place the decimal separator where you want it so that sums look like what you’re accustomed to.

In case you find this is a lot of fuss, the alternative is to use ad-hoc accounting dedicated applications. But check that the developers did things correctly, i.e. didn’t use IEEE-754 but implemented multi-precision arithmetic. Decades ago, the COBOL language was invented to address all these accounting issues (because in the end there are legal and tax mandatory rules to abide by). In COBOL, you could require that all computations be done in decimal mode to stick as nearly as possible to real life constraints. But even with this feature, you could botch your results.

To show the community your question has been answered, click the ✓ next to the correct answer, and “upvote” by clicking on the ^ arrow of any helpful answers. These are the mechanisms for communicating the quality of the Q&A on this site. Thanks!

1 Like

Note also that there’s no “universal integer mode”. There’s an integer mode of some programming language; an integer mode of some ALU; and - suddenly integer modes of accounting rules (which state e.g. what to do when you calculate percentages, and need not to be consistent there even with themselves - they may apply one way when you calculate taxes, and another for other calculations, and even for different taxes, inside one legislation, not mentioning differences between countries).

So dedicated accounting softwares “do things correctly” because they are working in the specific known constraints. There’s no single universal rule, and hence no silver bullet for general-purpose software like Calc.

@ajlittoz: :slight_smile: :slight_smile: :slight_smile:
thank you very much! i need additional ideas and and that other people ‘poke around in mine …’

  • integer arithmetik: yes, did so sometimes,have seen even banks switching to ‘cent amounts’ in reporting,
  • all is integer: no, once you reach ‘uneven fractions’ (working 9 hours for 100 bucks, bucks per hour?, 19% VAT) you need either fractional numbers, or approximations with e.g. decimal or binary values,
  • target: is not! to get my special calculations solved, but to improve calc in general in a way that it produces less irritation for ‘normal’ users with ‘normal’ math knowledge, or ‘better results for a wide range of different tasks’,
    @mikekaganski: yes, that’s additional problems, my target is not ‘correct accounting (for a special country)’ but ‘correct math’, once we get that everybody can apply it to his legal rules as neccessary,

As long as you remain inside the “validity domain” of your [arithmetic] assumptions, Calc delivers fairly good results. The “validity domain” is voluntarily not defined because it depends on numerous factors related to your data and intent.

Calc always does “correct math” over the subset of Q (the set of all fractions) corresponding to the IEEE-754 range. This subset has well known numerical analysis properties, some of which are not obvious and intuitive for the majority of users (and, yes, this leads to irritation and frustration). For example, the 4 arithmetic operations are not internal laws over the set (results cannot always be represented by a member of the set).

In your example of 9 hrs for 100€, read what I wrote about division. For 19%, do not use the %-unit, by write 19 followed by a division by 100. Yes, this is tedious. But a sheet may have hundreds rows and thousand columns. You can use “technical” cells (later made 0-size) to convert the values to integer and …

… base all your computations on the “technical cells”.

If your goal is ultimate accuracy, this is the cost. You can’t do without a careful analysis of your application and formulas (as in any production-quality application).

The problem is the long-standing belief that computers are always right. Right for integer calculation provided no overflow happened. Wrong for floating-point because precision is limited and you incur a radix conversion (we are used to think in 1/10 radix while IEEE-754 uses 1/2 radix). Note carefully I mentioned a fractional radix and not an integer radix because this is the way IEEE-754 is built! This is the origin of the difficulty because 5 (half of 10) and 2 are primes.

As a first approach, “normal” math knowledge is sufficient if you realise fraction 1/3 cannot be exactly represented in base 1/10. It is an infinite sequence 0.333… It will necessarily be truncated.

Advanced math is needed if you want to cope with floating-point with limited accuracy loss.

which is probably by definition : non significant

anywayz, hope you can stop trolling around, and if you ever have a useful additional point for actual users, feel free to consolidate at Frequently asked questions - Calc: Accuracy problem - The Document Foundation Wiki

but sometimes punches through and becomes relevant . However you are right, that fit’s better in devs or bugs forums. Was just another user asking here …

Troll (slang) - Wikipedia

→ desktop database front endBase | LibreOffice - Free and private office suite - Based on OpenOffice - Compatible with Microsoft

Well, it’s the LibreOffice component dealing with databases providing a frontend of forms and reports. Every database differentiates between integers, fixed decimals and floating point decimals.