View Single Post

   
  #2 (permalink)  
Old 04-08-2008, 06:17 PM
Mark A. Parsons
 
Posts: n/a
Default Re: Sybase/TransactSQL calculations

You're going to have to provide some more details about your environment
.... version of ASE ... how the values are stored in ASE (float, decimal,
hardcoded numbers in a multiplication equation, etc.).

---------------

On ASE 12.0.0.8 I get the same results as your excel spreadsheet numbers:

select -13.179101 * 1.437439 * .9998 * 200000000,
-7.655611 * 14.195282 * .988299 * 50000000
go

----------------------------------- -------------------------------------
-3788072986.3173064400000000 -5370098386.826276964900000000
(-3,788,072,986.3173) (-5,370,098,386.8263)

gcarter@acm.org wrote:

> Below are the results of AxBxCxD on 2 records 9609 and 9610 using
> Transact-SQL. As you can see for record 9609 the Excel result matches
> the Sybase result. However, the results differ for record 9610. Is this
> a precision/rounding issue?
>
> Thanks,
> GC
>
>
>
> RecordNo 9609 9610
> A -13.179101 -7.655611
> B 1.437439 14.195282
> C 0.9998 0.988299
> D 200000000 50000000
> Excel Results -3,788,072,986.3173 -5,370,098,386.8263
> Sybase Results -3,788,072,986.3173 -5,370,099,582.2354
> Diff 0.0000 1,195.4091
>

Reply With Quote