At present Calc cuts the number off at 12 decimal places. Is Calc capable of accepting 17 decimal-place numbers?
Thanks
You might like to read this FAQ and linked bookmarks first, Frequently asked questions - Calc: Accuracy problem - The Document Foundation Wiki
Then consider if a specialist mathematics program might be better suited to your task
Well, it’s really not a problem of accuracy but more of calc’s calculation prowess as it seems to be only able to calculate up to 16 digits in each column. It will accept additional decimal places with the “add decimal place” function, but it only accepts zeros which leads me to think it can only handle 16-digit calculations.
Here is one of the links that I thought you might have looked at, Numeric precision in Microsoft Excel - Wikipedia
consider if a specialist mathematics program might be better suited to your task
2 Likes
That’s not anything to do with columns or with Calc. Like all current “office” software Calc uses the FPU (FloatingPointUnit) of the standardised CPU (CentralProcessingUnit) it has access to. For this purpose the numbers are represented neither decimal nor with an arbitrary number of bits, but in standardised 64-bit format as given in IEEE 754 ‘Double’.
And handling numbers this way causes unavoidable limitations in accuracy which are taken in account by the process of converting numeric contents and results to “human readable” decimal representation for the display.
You may add decimal places to the displayed format, but beyond the reasonably rounded value of up to 15 significant digits, you will get shown zeroes.
1 Like
No, what actually happens is that it accepts / displays 15 significant decimals. This is done to decrease users confusion, when they discover that the calculations are inexact.
But all calculations use all 53 bits of significance (16-17 decimals) of 64-bit IEEE 754 number.
Note that it is not guaranteed that there are 17 correct decimals in any number representable in double-precision binary number. Sometimes only 16 decimals are correct.
Is this a question about mere input or is it about arithmetics? Calc accepts any number of decimals when you enter them as text.
Well, both actually. Like when you simply put two numbers 1 and 2 on the same column, then grab 1’s cell first, hold the grab, and scroll down to generate 3,4,5, and so forth. The problem is when I input more than 15 digits in the cell, it gets cut off which causes the subsequently generated numbers to be off by 3 cells. I’m trying to generate longitudinal data in 0.000120563271604933° increments from 0 to 360 which produces about 3 million cells. Can you make the LibreOffice calc accept this number? I thank you in advance for your interest.
Given that you are involving pi, why stop at only 17 digits?
Don’t you just need a grid 360 columns by 8294(.4 ?) rows labelled by whatever sector of a circle?
Or just use radians and never resolve pi
Why should you if you need at most 2985984 multiples of your tiny special unit?
Are you aware of the fact that at this resolution you still have a step of >13 m along the equator? This isn’t sufficient for recent GeoData systems.
At least I now may be able to understand this a little bit better. It shouldn’t be due to IEEE 754, but to a specific limitation applied to auto-IncrementDecrement in fill processes with textual content ending with decimal digits. However, from my excpereience this special limit is 10 digits + 1 carry for textual content. For actual whole numbers there is no such special limit.
Might it be that you are in a trap described in detail here?
3 million cells, or do you need this number of rows?
This old thread got a revival.
Well, if we take the subject question literally, the answer is very simple: Enter it as text.
Even very old versions will accept 65535 characters (digits, and possibly a decimal separator) in a single cell. With recent versions much more.
The intention to calculate with monster numbers is the challenge.
60 or more years ago there were no FPU and no standardised systems. Some families of computers from that time had the ability to calculate with decimal digits (which were, however, coded binary) like a primary school child, though a bit faster.
A simple BCD-calculation is exemplified in this spreadsheet (34.0 KB).