how to handle number values with high decimal place count

By mambrus · 2022-03-06

Description

Hey guys, I have values with up to 8 decimal places e.g. 292.77321149
and I want to dump them into a text file

----------------
example:

a = 292.77321149
newTxtFilePath = "C:\\Users\\marku\\Desktop\\tmp_output_2.txt"
newTxtFile = createFile newTxtFilePath
format "%\n" a to:newTxtFile

----------------
result in textfile:
292.773

----------------
how can I get exact value to my textfile?

Thanks!

Comments (4)

Swordslayer · 2022-03-09

Use doubles, not single-precision floats:

x = 292.77321149d0
formattedPrint x format: ".15g"
"292.77321149"

IDEA

mambrus · 2022-03-07

lets assume I have the values stored as strings

e.g.

a = "138.22182823"
b = "221.83117712"

... I assume I could move the "." inside the string

so I get

a = "138,221,828.23"
b = "221,831,177.12"

instead ...

now if I convert those to float .. I would expect to get the right result if adding both
BUT ... (i just tryed it) ... the result is wrong... why?

Sample Code:
--------------------------
a = "138221828.23"
b = "221831177.12"

c = a as float + b as float
-- expected result:
-- 138221828,23 + 221831177,12 = 360053005,35

formattedPrint c format: "9.9g"

some more thoughts

mambrus · 2022-03-07

I also found this:

"Note:
The range of valid Integers in MAXScript is -2,147,483,648 to 2,147,483,647. If you perform calculations that result in integers outside of this range, you will get integer overflow errors that are not detected by MAXScript. You must take care in designing your code to prevent or detect this overflow yourself. The result of an overflow is typically a large number of the wrong sign. Dividing an integer by 0 will result in a MAXScript system exception.

A Float in MAXScript has an absolute value range of 1.18E-38 to 3.40E38 with a precision of one part in 1.0E7. If you perform calculations that result in floats with an absolute value less than this range, the result will be stored as 0.0. If you perform calculations that result in floats with an absolute value larger than this range, the result will be stored as a special value that represents infinity, 1.#INF . Adding, subtracting, or multiplying a number by 1.#INF results in a value of 1.#INF . Dividing a number by 1.#INF results in a value of 0.0. Dividing 0.0 by 0.0 or 1.#INF by 1.#INF , multiplying 1.#INF by 0, or 1.#INF from 1.#INF results in a special value that represents an indeterminate number, - 1.#IND . Adding, subtracting, multiplying, or dividing a number by - 1.#IND results in a value of - 1.#IND .

When you display or print a Float, MAXScript prints the value to the 6th significant digit. The following table shows examples of values stored to a MAXScript variable, the value as stored, and the value as displayed."

Here: https://help.autodesk.com/view/MAXDEV/2022/ENU/?guid=GUID-B57EA575-DCDE-…

which tells me that in general this is not possible ... BUT ...

some thoughts

mambrus · 2022-03-07

so from my understanding already the first line does not realy work since I assume a float wont hold up to 8 decimal places, right?

how do I get such a precise number processed?

I also read about formattedPrint ... but even with this (and getting up to 8 digits) ... the result of my calculations (I am adding up such values) is wrong - so the values must have been processed/altered/reduced in digits .. while being added

any ideas?