You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Make cast from float/double to decimal compatible with Spark #1371
@coderfender go ahead.
Also, from my notes -
The incompatibility comes from rounding differences between Comet and Spark. Comet uses the binary representation of the numbers and sometimes these numbers are not represented exactly in IEEE 754. (e.g. 1.005 is actually 1.00499999999999989). Spark represents these as string. With rounding Comet ends up with 1.00, Spark ends up with 1.01.
This happens for a very small set of numbers. For deterministic data however, it is a correctness issue.
String representation comes at the cost of performance.
So now the tradeoff is - faster, but sometimes not exact OR slower but always exact.
I would like to have both, and default to the faster version, but others might have different opinions.
Activity
andygrove commentedon May 6, 2026
@coderfender is this issue still valid?
coderfender commentedon May 7, 2026
Yeah , this still seems like an issue . We have a commented test commented out as well
coderfender commentedon May 7, 2026
@parthchandra I can take stab at it unless you want to ?
parthchandra commentedon May 7, 2026
@coderfender go ahead.
Also, from my notes -
The incompatibility comes from rounding differences between Comet and Spark. Comet uses the binary representation of the numbers and sometimes these numbers are not represented exactly in IEEE 754. (e.g. 1.005 is actually 1.00499999999999989). Spark represents these as string. With rounding Comet ends up with 1.00, Spark ends up with 1.01.
This happens for a very small set of numbers. For deterministic data however, it is a correctness issue.
String representation comes at the cost of performance.
So now the tradeoff is - faster, but sometimes not exact OR slower but always exact.
I would like to have both, and default to the faster version, but others might have different opinions.
fix: make native cast from float/double to decimal compatible with Sp…
fix: make native cast from float/double to decimal compatible with Sp…