Icon View Thread

The following is the text of the current message along with any replies.
Messages 1 to 8 of 8 total
Thread Unusual Rounding Behavior
Wed, Dec 16 2020 12:30 PMPermanent Link

Kevin Koehne

Has anyone experienced the Round function returning different values on different machines?

I've run this query in a variety of environments and one server returns a different result.

Select 110.00 * 0.1025 noRound,
      Round(110.00 * 0.1025, 2) roundNoCast,
      Round(Cast(110.00, Decimal(19,2)) * 0.1025, 2) roundCastD2,
      Round(Cast(110.00, Decimal(19,2)) * Cast(0.1025, Decimal(19,4)), 2) roundCastD2_D4,
      Round(Cast(110.00, Decimal(19,2)) * Cast(0.1025, Float), 2) roundCastD2_F

The values I get generally are:
noRound = 11.275
roundNoCast = 11.27
roundCastD2 = 11.28
roundCastD2_D4 = 11.2800
roundCastD2_F = 11.28

But on one customer's server the result for that same query is different (highlighted with *)
noRound = 11.275
roundNoCast = 11.27
* roundCastD2 = 11.27
roundCastD2_D4 = 11.2800
* roundCastD2_F = 11.27

They are running an older version (2.27.0.1) and I will try to update them. But I have tried that query with that version and I always get the first results.

I noticed it on a table where a Decimal(19,2) field is multiplied by a Float field (which is the scenario that is returning different value). I can't change the type on the Float because I need more than 4 decimal places.

Any thoughts?

Thanks,
Kevin
Wed, Dec 16 2020 3:13 PMPermanent Link

Raul

Team Elevate Team Elevate

On 12/16/2020 12:30 PM, Kevin Koehne wrote:
>
> The values I get generally are:
> noRound = 11.275
> roundNoCast = 11.27
> roundCastD2 = 11.28
> roundCastD2_D4 = 11.2800
> roundCastD2_F = 11.28
>
> But on one customer's server the result for that same query is different (highlighted with *)
> noRound = 11.275
> roundNoCast = 11.27
> * roundCastD2 = 11.27
> roundCastD2_D4 = 11.2800
> * roundCastD2_F = 11.27
>

Not sure it helps but I'm getting the 2nd set of results (11.27 for
roundcastD2 and roundCastD2_F) on my main Win10 system using  edb
manager 2.33 b2.

Other than using ryzen cpu it's a normal win 10 install and set to US
regional settings (even though i'm in Canada)

Raul
Thu, Dec 17 2020 8:38 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

Snap using EDBManager on a random table

W7x64
i7 16GB RAM
ElevateDB 2.33b2


Roy Lambert
Thu, Dec 17 2020 8:53 AMPermanent Link

Kevin Koehne

Thanks @Raul - I wonder if it is a chipset difference

@Roy - Thanks for the system info. Did you get the same results as I did or Raul?
Thu, Dec 17 2020 10:25 AMPermanent Link

Roy Lambert

NLH Associates

Team Elevate Team Elevate

Kevin


Raul's

Roy Lambert
Thu, Dec 17 2020 1:53 PMPermanent Link

Raul

Team Elevate Team Elevate

On 12/17/2020 8:53 AM, Kevin Koehne wrote:
> Thanks @Raul - I wonder if it is a chipset difference

AFAIK all of this is really just float point calc on CPU so any
differences would be either that or maybe some windows settings (set on
underlying hardware).

Considering we're getting the 2nd result so far (Roy and me at least) i
wonder if the 2nd is the more common answer !?


Raul








Thu, Dec 17 2020 2:35 PMPermanent Link

Kevin Koehne

It is definitely environmental.

I've installed the latest server on my machines and I still get 11.28 for roundCastD2.

I've tried it on at least 5 different computers (in the office and other clients) and got 11.28. Only on one computer did I see 11.27.

They've all been Intel CPUs running Win10 or WinServer 2016.
Thu, Dec 17 2020 4:00 PMPermanent Link

Raul

Team Elevate Team Elevate

On 12/17/2020 2:35 PM, Kevin Koehne wrote:
> It is definitely environmental.
>
> I've installed the latest server on my machines and I still get 11.28 for roundCastD2.
>
> I've tried it on at least 5 different computers (in the office and other clients) and got 11.28. Only on one computer did I see 11.27.
>
> They've all been Intel CPUs running Win10 or WinServer 2016.

I tried it on one of our test servers (Win 2012R2 VM with Intel Zeon
CPU) - 11.27 still for me

Raul
Image