Total of count queries differs for each dive while using the same data

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
Report Content as Inappropriate

Total of count queries differs for each dive while using the same data

This post has NOT been accepted by the mailing list yet.
I get different total counts when I try to sum up the path values for different dives. The returned values from the API are in the float format which keeps precision of maximum 6 figures and omits the remaining part of the result by using exponent. I have a simple test data with 2 categories having 3 and 50 subcategories respectively. Total Count is about 500,000,000,000. Here are the different queries I used for the same data:


As you can see, the sum of cat1 is 500,062,000,000 and the sum of cat2 is 500,062,380,000

Difference is 380,000 because of the precision lost in the query results. Even with the better precision, the second result is also wrong. The correct result should be 500,062,346,584

I know the difference is not too big compared to the total; however, it really creates a discrepancy.

The same error happens when you are looking at the map total on the upper right corner of the web viewer. If I look at the same area on different zoom levels, the total is also different because the number of tiles to sum differs for queries in each zoom level. Here is the total results for three different zoom levels for the same area.

For the total on the map, the differences are even bigger.

How can I solve this issue? Is there a way to get the exact number for the queries without rounding the numbers? Does the API support such a thing?