Ask Your Question

Calc Basic macro integer arithmetic problem [closed]

asked 2016-06-07 15:41:41 +0200

Ajohn gravatar image

updated 2016-06-07 15:45:07 +0200

I'm dealing with prime number fractions and have noticed that Calc Basic isn't really using integer. It's rounding if I do a divide for instance rather than truncation. It's using some sort of rounding process. I also then noticed that some integer functions are stated to return double so it looks like pure integer is never used under the hood.

I wonder if there are any compiler options to prevent the rounding or even better force true integer. Mod seems to work ok but as there is no quotient that I am aware of so I have to use divide.

I chose the integer way as the macro has to loop through 4,000 numbers and I can't check all of them. If done in pure integer all numbers have to be ok just on the basis of one being correct. Float has been known to cause problems with what I am doing.


edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by Alex Kemp
close date 2020-10-29 16:56:48.899029


Well I didn't expect an answer but feel the lack of true integer variables is disgusting.

Ajohn gravatar imageAjohn ( 2016-06-11 17:13:21 +0200 )edit

As opposed to the spreadsheet cells where any numerical value is IEEE Double, LibreOffice BASIC has integer types. What you may grumble about is that integer arithmetic isn't supported to the degree you would like. Btw: the operator @hasklo mentioned should be equivalent to Quotient(Numerator, Denominator).
Generally machine arithmetic does not support unlimited precision for integer numbers. If you need it, you have to resort to mathematical software. Try the free 'Maxima' e.g.

Lupp gravatar imageLupp ( 2018-02-08 12:15:35 +0200 )edit

1 Answer

Sort by » oldest newest most voted

answered 2018-02-08 11:37:33 +0200

hasklo gravatar image

I know it's been really long since you asked this question... but maybe this can help someone with similar question.

Andrew Pitonyak answered this question in his book " Macros Explained" under the section "Integer division". You can download a PDF version here: []...

Basically, the "/" operator returns double. If you set it to an integer, it gets rounded as you have found. There is another operator specifically for integer division: "\" (from left to right slash), which will truncate the result.

dim iMe as Integer
iMe = 3\4    ' --> iMe is 0
edit flag offensive delete link more


thank you!! I thought I was going nuts.

CaveMan gravatar imageCaveMan ( 2020-06-21 13:54:34 +0200 )edit

Question Tools

1 follower


Asked: 2016-06-07 15:41:41 +0200

Seen: 706 times

Last updated: Feb 08 '18