Community Assistance Required - Please Help :)

[madpeople]

I would assue it's related to the precision intermediary results are stored in the registers in the CPU.

If this isn't sorted by the end of the week I'll whip up a java, but until then i'm a bit busy.

(see strictfp)

[sgt_baker]

QUOTE (madpeople @ Dec 15 2008, 06:47 PM) I would assue it's related to the precision intermediary results are stored in the registers in the CPU.

If this isn't sorted by the end of the week I'll whip up a java, but until then i'm a bit busy.

I'm suspecting something of this nature. I was going to further inquire as to machine config for anyone posting values which were significantly off the mark.

B

[cashto]

Well, since you haven't told us what methods you are using to calculate these numbers, and what results you're getting with them, I can only spectulate ...

Since all the numbers involved are roughly the same magnitude, I don't think you're running into the non-commutative nature of machine floating-point math (e.g., the fact that if you add lots of little numbers up first and then add it to a big number, you get a different result than if you took the big number first and added all the little numbers up after it. The reason being is rounding errors causing loss of precision).

Instead what I suspect is that in computing environment A, you're using 32-bit floats (~7 digits of precision) and in environment B you're using 64-bit doubles (~15 digits of precision).

[ogorass]

result = 277.90524170467029

Using C#.NET (.NET ver. 3.5, VS 2k8), Intel Core 2 Duo (T7250).

[deschain]

Result = 10

Using: fingers

[sgt_baker]

QUOTE (cashto @ Dec 15 2008, 07:08 PM) Well, since you haven't told us what methods you are using to calculate these numbers, and what results you're getting with them, I can only spectulate ...

Since all the numbers involved are roughly the same magnitude, I don't think you're running into the non-commutative nature of machine floating-point math (e.g., the fact that if you add lots of little numbers up first and then add it to a big number, you get a different result than if you took the big number first and added all the little numbers up after it. The reason being is rounding errors causing loss of precision).

Instead what I suspect is that in computing environment A, you're using 32-bit floats (~7 digits of precision) and in environment B you're using 64-bit doubles (~15 digits of precision).

ASGS result: 277.90524170467012
Everyone else: 277.90524170467029

The ASGS SQL Server definitely uses 64-bits for its FLOAT datatype. The error is minimal when performing the simple calculation listed above, but we have a load of much more hardcore maths running in AllegSkill, and one of the vital numbers is off by a whole 50% when compared to its correct value. I'm running and exact mirror of the ASGS AllegSkill code which renders correct results. As far as I can tell the problem *only* manifests on the ASGS box.

B

Edit: And the method was SELECT SUM(Mu * FractionPlayed)

Edit2: Although the ASGS value is precise to 15 digits

[CronoDroid]

277.905242

Google Search!

[cashto]

QUOTE (sgt_baker @ Dec 15 2008, 11:46 AM) ASGS result: 277.90524170467012
Everyone else: 277.90524170467029

The ASGS SQL Server definitely uses 64-bits for its FLOAT datatype. The error is minimal when performing the simple calculation listed above, but we have a load of much more hardcore maths running in AllegSkill, and one of the vital numbers is off by a whole 50% when compared to its correct value. I'm running and exact mirror of the ASGS AllegSkill code which renders correct results. As far as I can tell the problem *only* manifests on the ASGS box.

B

Edit: And the method was SELECT SUM(Mu * FractionPlayed)

Edit2: Although the ASGS value is precise to 15 digits

According to this IEEE calculator:

277.90524170467012 = 4071 5E7B DEB9 C889
277.90524170467029 = 4071 5E7B DEB9 C88C

So the error you are seeing is 3 ulps. That could easily be accounted for due to the non-associative property of machine floating-point.

In English, everyone's elses compiler is doing the math exactly as you asked for it -- left to right. But since you're using SQL, under the hood the database might be fetching and/or computing the numbers in some random order.

My suggestion is that if an error as small as 3 ulps in an intermediate calculation results in a 50% change in the final result, then whatever algorithm you are using is highly succeptible to rounding error; you need to do some numerical analysis (for example, using interval arithmetic) to determine where you are losing precision.

Edit: I meant non-associative, not non-commutative.

[parcival]

277.9052417046702829988978297443
Windows calc.

[Grzegt]

277,90524170467 open office calc

277.90524170467 activeperl 5.8.8

AMD Athlon XP 2000+, win XP