Login ProductsSalesSupportDownloadsAbout |
Home » Technical Support » ElevateDB Technical Support » Support Forums » ElevateDB General » View Thread |
Messages 1 to 8 of 8 total |
Unusual Rounding Behavior |
Wed, Dec 16 2020 12:30 PM | Permanent Link |
Kevin Koehne | Has anyone experienced the Round function returning different values on different machines?
I've run this query in a variety of environments and one server returns a different result. Select 110.00 * 0.1025 noRound, Round(110.00 * 0.1025, 2) roundNoCast, Round(Cast(110.00, Decimal(19,2)) * 0.1025, 2) roundCastD2, Round(Cast(110.00, Decimal(19,2)) * Cast(0.1025, Decimal(19,4)), 2) roundCastD2_D4, Round(Cast(110.00, Decimal(19,2)) * Cast(0.1025, Float), 2) roundCastD2_F The values I get generally are: noRound = 11.275 roundNoCast = 11.27 roundCastD2 = 11.28 roundCastD2_D4 = 11.2800 roundCastD2_F = 11.28 But on one customer's server the result for that same query is different (highlighted with *) noRound = 11.275 roundNoCast = 11.27 * roundCastD2 = 11.27 roundCastD2_D4 = 11.2800 * roundCastD2_F = 11.27 They are running an older version (2.27.0.1) and I will try to update them. But I have tried that query with that version and I always get the first results. I noticed it on a table where a Decimal(19,2) field is multiplied by a Float field (which is the scenario that is returning different value). I can't change the type on the Float because I need more than 4 decimal places. Any thoughts? Thanks, Kevin |
Wed, Dec 16 2020 3:13 PM | Permanent Link |
Raul Team Elevate | On 12/16/2020 12:30 PM, Kevin Koehne wrote:
> > The values I get generally are: > noRound = 11.275 > roundNoCast = 11.27 > roundCastD2 = 11.28 > roundCastD2_D4 = 11.2800 > roundCastD2_F = 11.28 > > But on one customer's server the result for that same query is different (highlighted with *) > noRound = 11.275 > roundNoCast = 11.27 > * roundCastD2 = 11.27 > roundCastD2_D4 = 11.2800 > * roundCastD2_F = 11.27 > Not sure it helps but I'm getting the 2nd set of results (11.27 for roundcastD2 and roundCastD2_F) on my main Win10 system using edb manager 2.33 b2. Other than using ryzen cpu it's a normal win 10 install and set to US regional settings (even though i'm in Canada) Raul |
Thu, Dec 17 2020 8:38 AM | Permanent Link |
Roy Lambert NLH Associates Team Elevate | Snap using EDBManager on a random table
W7x64 i7 16GB RAM ElevateDB 2.33b2 Roy Lambert |
Thu, Dec 17 2020 8:53 AM | Permanent Link |
Kevin Koehne | Thanks @Raul - I wonder if it is a chipset difference
@Roy - Thanks for the system info. Did you get the same results as I did or Raul? |
Thu, Dec 17 2020 10:25 AM | Permanent Link |
Roy Lambert NLH Associates Team Elevate | Kevin
Raul's Roy Lambert |
Thu, Dec 17 2020 1:53 PM | Permanent Link |
Raul Team Elevate | On 12/17/2020 8:53 AM, Kevin Koehne wrote:
> Thanks @Raul - I wonder if it is a chipset difference AFAIK all of this is really just float point calc on CPU so any differences would be either that or maybe some windows settings (set on underlying hardware). Considering we're getting the 2nd result so far (Roy and me at least) i wonder if the 2nd is the more common answer !? Raul |
Thu, Dec 17 2020 2:35 PM | Permanent Link |
Kevin Koehne | It is definitely environmental.
I've installed the latest server on my machines and I still get 11.28 for roundCastD2. I've tried it on at least 5 different computers (in the office and other clients) and got 11.28. Only on one computer did I see 11.27. They've all been Intel CPUs running Win10 or WinServer 2016. |
Thu, Dec 17 2020 4:00 PM | Permanent Link |
Raul Team Elevate | On 12/17/2020 2:35 PM, Kevin Koehne wrote:
> It is definitely environmental. > > I've installed the latest server on my machines and I still get 11.28 for roundCastD2. > > I've tried it on at least 5 different computers (in the office and other clients) and got 11.28. Only on one computer did I see 11.27. > > They've all been Intel CPUs running Win10 or WinServer 2016. I tried it on one of our test servers (Win 2012R2 VM with Intel Zeon CPU) - 11.27 still for me Raul |
This web page was last updated on Thursday, May 23, 2024 at 07:54 PM | Privacy PolicySite Map © 2024 Elevate Software, Inc. All Rights Reserved Questions or comments ? E-mail us at info@elevatesoft.com |