techSetPrecision
techSetPrecision(
x_digits
)
=> t / nil
Description
Sets the precision, or number of digits following the decimal point, to be used in floating-point calculations.
This function treats a real number as a float (double) number and performs the following checks.
-
If absolute value of number less than equal to
1/10^n(precision), then the same number is returned. - If absolute value of number greater than precision, then it is rounded off to the precision digit.
Arguments
|
Number of digits following the decimal point in floating-point numbers. |
Value Returned
|
The precision was not set because the specified precision value was invalid. |
Example
techSetPrecision(6)
Sets the precision for floating-point numbers to 6.
Return to top