When entering calculation formulas into numerical fields, results are incorrect in some cases. In the following example, inventory item purchase and sales prices are used for illustration. But the bug seems to exist anywhere calculations are permitted.
For the illustration, calculations of [100 * X] were entered.
- When X was 1.085 or below, the result was correct.
- When X was 1.0851–1.152, results were off in the 14th decimal place, sometimes higher and sometimes lower than the exact result should have been. This suggests the problem is related to floating point arithmetic.
- When X was greater than 1.152, results were again correct.
Specific values of X used in the illustration were as shown in the table below, arranged to match the layout of the output screen shot below:
|Column 1||Column 2|
Manager’s calculated values of [100 * X] are shown below:
While differences at the 14th decimal place might seem insignificant, and do not matter much on a tab listing, this problem also shows up when calculations are used on transaction forms. For example, a calculated unit price of [217.12 * 1.1] yields:
The exact unit price is 238.832. (Of course, in this example, an error in total amount would not show up unless the quantity was very large.)
Testing did not reveal another range where calculation results are erroneous. But that does not mean other ranges do not exist, only that they were not found.