Ask Your Question
0

Calc Basic macro integer arithmetic problem

asked 2016-06-07 15:41:41 +0200

Ajohn gravatar image

updated 2016-06-07 15:45:07 +0200

I'm dealing with prime number fractions and have noticed that Calc Basic isn't really using integer. It's rounding if I do a divide for instance rather than truncation. It's using some sort of rounding process. I also then noticed that some integer functions are stated to return double so it looks like pure integer is never used under the hood.

I wonder if there are any compiler options to prevent the rounding or even better force true integer. Mod seems to work ok but as there is no quotient that I am aware of so I have to use divide.

I chose the integer way as the macro has to loop through 4,000 numbers and I can't check all of them. If done in pure integer all numbers have to be ok just on the basis of one being correct. Float has been known to cause problems with what I am doing.

John

edit retag flag offensive close merge delete

Comments

Well I didn't expect an answer but feel the lack of true integer variables is disgusting.

Ajohn gravatar imageAjohn ( 2016-06-11 17:13:21 +0200 )edit
1

As opposed to the spreadsheet cells where any numerical value is IEEE Double, LibreOffice BASIC has integer types. What you may grumble about is that integer arithmetic isn't supported to the degree you would like. Btw: the operator @hasklo mentioned should be equivalent to Quotient(Numerator, Denominator).
Generally machine arithmetic does not support unlimited precision for integer numbers. If you need it, you have to resort to mathematical software. Try the free 'Maxima' e.g.

Lupp gravatar imageLupp ( 2018-02-08 12:15:35 +0200 )edit

1 Answer

Sort by » oldest newest most voted
2

answered 2018-02-08 11:37:33 +0200

hasklo gravatar image

I know it's been really long since you asked this question... but maybe this can help someone with similar question.

Andrew Pitonyak answered this question in his book "OpenOffice.org Macros Explained" under the section "Integer division". You can download a PDF version here: [http://www.pitonyak.org/OOME_3_0.pdf]...

Basically, the "/" operator returns double. If you set it to an integer, it gets rounded as you have found. There is another operator specifically for integer division: "\" (from left to right slash), which will truncate the result.

dim iMe as Integer
iMe = 3\4    ' --> iMe is 0
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2016-06-07 15:41:41 +0200

Seen: 274 times

Last updated: Feb 08 '18