You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg years/months/days/hours differ from Iceberg for some pre-1970 timestamps #6426
#5638 added native kernels for Iceberg's years, months, days and hours functions, and they run by default. #5773 also routes these functions to the native kernels when Iceberg's SQL extensions rewrite them to ApplyFunctionExpression.
The kernels compute a true floor. Iceberg's Java DateTimeUtil returns one unit less for some timestamps: a negative timestamp whose microseconds are 999999 at a unit boundary. Iceberg's Spark functions call DateTimeUtil, so Spark returns Iceberg's value, and it is also the value Iceberg partitions by.
In 1.0.0 these functions ran in Spark, so this is a regression in 1.1.0. It reproduces on 1.1.0-rc1, and main has the same kernels.
Steps to reproduce
With an Iceberg catalog ice, the session timezone set to UTC, and a table t holding TIMESTAMP '1969-01-01 00:00:00.999999', run:
SELECTice.system.years(ts), ice.system.months(ts), ice.system.days(ts), ice.system.hours(ts) FROM t
With Iceberg's SQL extensions installed, a filter such as WHERE hours(ts) = -2 goes through the same kernels.
Expected behavior
Spark with Iceberg, and Comet 1.0.0, return -2, -13, 1968-12-31, -8761. Comet 1.1.0-rc1 returns -1, -12, 1969-01-01, -8760.
For a table with 1969-12-31 23:00:00.999999 and 1969-12-31 22:30:00, WHERE hours(ts) = -2 returns both rows in Spark and only the second on rc1. TIMESTAMP_NTZ inputs behave the same way.
Additional context
For negative input, DateTimeUtil.convertMicros builds the instant from two parts: floorDiv(micros, 1_000_000) seconds and floorMod(micros + 1, 1_000_000) microseconds. It then subtracts one from the unit count. At these boundaries that undercounts by one. Iceberg 1.5.2 through 1.11.0 all behave this way.
The native kernels in native/spark-expr/src/iceberg_funcs/temporal.rs need to mirror it to agree with Iceberg. The inputs that trigger it are rare, but the result is wrong silently.
Found by the 1.1.0 regression audit (#6399) and tracked in #6402.
Activity
fix: match Iceberg's rounding for pre-1970 timestamps in native years…
docs: drop the six regressions that branch-1.1 now fixes from the rel…