You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native date functions and date-to-string cast return NULL or fail for dates outside chrono's NaiveDate range #6843
Spark DateType covers any epoch day that fits in Int (years from about -5877641 to 5881580). Several native Comet date kernels convert the value through chrono's NaiveDate, which only covers years -262143 to 262142. For dates outside that range, these kernels return NULL where Spark returns a value, or fail the query.
Such dates reach native execution from a Parquet date column and from native make_date (since #5443).
Steps to reproduce
SETspark.sql.session.timeZone=UTC;
CREATETABLEt(d date) USING parquet;
INSERT INTO t SELECT date_from_unix_date(x)
FROMVALUES (95026237), (-96465293), (108853388), (18262) AS v(x);
-- 262143-01-01, -262144-12-31, 300000-06-15, 2020-01-01SELECT unix_date(d), year(d), month(d), dayofmonth(d) FROM t;
SELECT unix_date(d), cast(d AS string) FROM t;
SELECT unix_date(d), last_day(d) FROM t;
The 2020-01-01 row matches Spark in every case. For the other three rows:
the field value (e.g. year = 262143, -262144, 300000)
NULL
trunc(d, 'MM'), next_day(d, 'Mon')
a date
NULL
cast(d AS string)
e.g. +262143-01-01
query fails: Cast error: Failed to convert 95026237 to temporal for Date32
last_day(d)
a date
query fails: Spark last_day: Unable to convert days value 108853388 to date
dayofweek, add_months, date_add, unix_date and date comparisons already match Spark for these values.
Expected behavior
These expressions should return the same values as Spark for every date that Spark DateType can hold, or fall back to Spark when the native kernel cannot represent the value.
Additional context
Casting such dates to TIMESTAMP panics, which is tracked separately in #5456. date_format is not listed because Spark itself throws ArithmeticException: long overflow for year 300000.
Activity
andygrove commentedon Oct 10, 2026
We should move to jif. I created the first arrow-rs PR towards this: apache/arrow-rs#11482