# calc funny result with cell formatted as 'fraction'

hi @all,

playing around with numbers and formatting i keyed in =3333333333/100 in a cell and formatted it as fraction with the format-string '?/???' (i wanted to display '3333333333/100' to come near to integer math),

displayed result: **-961633963/100** ?!?!

bug? limitation? 32-bit math? or did i do something wrong?

behaviour applies to values which stay with the denominator 100, others which can be reduced to 50-th or 25-th display correct,

same behaviour for plenty other values, seems to start with 2147483648/100,

happened in 7.1 from 2020-10-04, 6.1.6.3 same behaviour ...

formatting to '?/100' fails for all values above 2147483647, formatting to '?/50' holds till double that value, i suspect fail results from enumerator being above 2147483647 ... 2^{31}-1 holds and 2^{31} fails

[edit]

edited to provide some info for @Lupp, actually it is OT for this question, but where the discussion is going so well ...

question 1 is how and why do you or calc assume double floats below a certain threshold to be integers, neglecting that they may be rounded representations for fractions, and deny the same assumption for bigger values as it was taught in the cited text: 'So any mathematical operation that would have resulted in an odd number in this range will instead be rounded to one of the even numbers around it. But the result will still be an integer.' is there any 'external clue' for 'small' doubles that they are! integers and not rounded fractions which i'd miss?

my comment 'small hope for change in near future' references to @erAck's explanation that (assumed) integers are differently handeled for display than decimal fractions or 'big integers' which calc doesn't assume integer,

you requested a sample sheet - ok, - i can deliver - click to download / open - ,

i would like my favorite spreadsheet to deliver a little less nonsensical results than those marked yellow and red in the attached table ... meaningful intelligent rounding - yes, exaggerated falsifying rounding to hide errors which hopefully have been solved in the meantime - no, and even less so if it is only applied to successive values alternately,

i have marked the desired values in green next to it ...

P.S. please be indulgent if i got lost in details, it is about the basic concept ...

[/edit]

Ad

"question 1 is..."I don't. You may interpret the behavior of Calc this way, but it will not be fruitful. It's the other way round: Double FP-numbers with an effective dyadic exponent >=53 are

supposedtonotbe integer because therangebeing represented by the exactly same bit pattern contains more than one integer number, and there is no information available which one should be preferred. Concerning numbers with effective dyadic exponents 0<=e<=52 there is at least a realistic probability that they may be (in mind, not in sheet) be assured to be integer due to the ways they were calculated, and to the ideas behind the sheet formulas about which Calc can't know anything. In the context it may be useful to remember that the function INT() also returns aDoubleresult in every case, and that it simply is the ...(more)(Hope to find time tomorrow to comment on your claim concerning "nonsensical results". Cannot be sure.)