SEO

Below are the results of AxBxCxD on 2 records 9609 and 9610 using
Transact-SQL. As you can see for record 9609 the Excel result matches
the Sybase result. However, the results differ for record 9610. Is this
a precision/rounding issue?

Thanks,
GC



RecordNo 9609 9610
A -13.179101 -7.655611
B 1.437439 14.195282
C 0.9998 0.988299
D 200000000 50000000
Excel Results -3,788,072,986.3173 -5,370,098,386.8263
Sybase Results -3,788,072,986.3173 -5,370,099,582.2354
Diff 0.0000 1,195.4091

You're going to have to provide some more details about your environment
.... version of ASE ... how the values are stored in ASE (float, decimal,
hardcoded numbers in a multiplication equation, etc.).

---------------

On ASE 12.0.0.8 I get the same results as your excel spreadsheet numbers:

select -13.179101 * 1.437439 * .9998 * 200000000,
-7.655611 * 14.195282 * .988299 * 50000000
go

----------------------------------- -------------------------------------
-3788072986.3173064400000000 -5370098386.826276964900000000
(-3,788,072,986.3173) (-5,370,098,386.8263)

gcarter@acm.org wrote:

> Below are the results of AxBxCxD on 2 records 9609 and 9610 using
> Transact-SQL. As you can see for record 9609 the Excel result matches
> the Sybase result. However, the results differ for record 9610. Is this
> a precision/rounding issue?
>
> Thanks,
> GC
>
>
>
> RecordNo 9609 9610
> A -13.179101 -7.655611
> B 1.437439 14.195282
> C 0.9998 0.988299
> D 200000000 50000000
> Excel Results -3,788,072,986.3173 -5,370,098,386.8263
> Sybase Results -3,788,072,986.3173 -5,370,099,582.2354
> Diff 0.0000 1,195.4091
>

gcarter@acm.org wrote:

> Below are the results of AxBxCxD on 2 records 9609 and 9610 using
> Transact-SQL. As you can see for record 9609 the Excel result matches
> the Sybase result. However, the results differ for record 9610. Is this
> a precision/rounding issue?
>
> Thanks,
> GC
>
>
>
> RecordNo 9609 9610
> A -13.179101 -7.655611
> B 1.437439 14.195282
> C 0.9998 0.988299
> D 200000000 50000000
> Excel Results -3,788,072,986.3173 -5,370,098,386.8263
> Sybase Results -3,788,072,986.3173 -5,370,099,582.2354
> Diff 0.0000 1,195.4091

yes. if you expect a DBMS to do math, you must use exact
decimal column types, and for the Sybase table, you are
not.
Joe Weinstein at BEA Systems

I remember this is a bug from Sybase linux.
If the number is 50000000 or 500000000 and you use it in
multiplication, you will get the incorrect result, please check your
platform.

gcarter@acm.org wrote:
> Below are the results of AxBxCxD on 2 records 9609 and 9610 using
> Transact-SQL. As you can see for record 9609 the Excel result matches
> the Sybase result. However, the results differ for record 9610. Is this
> a precision/rounding issue?
>
> Thanks,
> GC
>
>
>
> RecordNo 9609 9610
> A -13.179101 -7.655611
> B 1.437439 14.195282
> C 0.9998 0.988299
> D 200000000 50000000
> Excel Results -3,788,072,986.3173 -5,370,098,386.8263
> Sybase Results -3,788,072,986.3173 -5,370,099,582.2354
> Diff 0.0000 1,195.4091