hi @all,
there are frequent questions in ‘ask’ and ‘bugs’ about missing accuracy in calculations, which often get the answer ‘it is regarding the limitations and “granularity” of limited length floating point numbers (IEEE 754) used in calc and most other spreadsheets to represent values, and by the small errors that occur when converting from decimal to binary representation, and vice versa, since the two systems partly cover the same values, but often do not’.
in an attempt to understand and visualize which errors evolve from that i’d construct a sheet to ‘recalculate’ an IEEE 754 number, it’s attached here: questions_about_fp_value_calculations_ori.ods
the questions are in the sheet, in short:
am i right? calc is using [edit] ‘16 bit’ was wrong, i meant ‘16 decimal digit’ [/edit] accuracy for integers, decimal fractions are shortened to 15 digit precision,
but there are more significant bits not evaluated by calc? question B and C in the sheet,
2^53 is rounded down by calc, regardless it has an exact IEEE 754 representaion? D in the sheet,
and a display error for one specific value, question E, i already filed a bug for that: https://bugs.documentfoundation.org/show_bug.cgi?id=131312
my ‘calculator’ holds for plenty samples of values in wikipedia, but not for:

1.0000000000000002, the smallest number > 1,

subnormals  not implemented,

(Min. normal positive double)  normally it should?

(Max. Double)  normally it should?
can calc do better or are this the limitations of the sheet?
i’d appreciate if somebody can recheck the sheet, and shed some light, either tell me where i’m wrong, or confirm that there are issues regarding the accuracy produced by calc.
reg.
b.
P.S. add. sheet with irritating results: questions_about_fp_value_calculations_2_ori.ods