Skip to content

Native date functions and date-to-string cast return NULL or fail for dates outside chrono's NaiveDate range #6843

Open

Description

@viirya

Describe the bug

Spark DateType covers any epoch day that fits in Int (years from about -5877641 to 5881580). Several native Comet date kernels convert the value through chrono's NaiveDate, which only covers years -262143 to 262142. For dates outside that range, these kernels return NULL where Spark returns a value, or fail the query.

Such dates reach native execution from a Parquet date column and from native make_date (since #5443).

Steps to reproduce

SET spark.sql.session.timeZone=UTC;
CREATE TABLE t(d date) USING parquet;
INSERT INTO t SELECT date_from_unix_date(x)
  FROM VALUES (95026237), (-96465293), (108853388), (18262) AS v(x);
-- 262143-01-01, -262144-12-31, 300000-06-15, 2020-01-01

SELECT unix_date(d), year(d), month(d), dayofmonth(d) FROM t;
SELECT unix_date(d), cast(d AS string) FROM t;
SELECT unix_date(d), last_day(d) FROM t;

The 2020-01-01 row matches Spark in every case. For the other three rows:

Expression Spark Comet
year, month, dayofmonth, dayofyear, quarter, weekofyear the field value (e.g. year = 262143, -262144, 300000) NULL
trunc(d, 'MM'), next_day(d, 'Mon') a date NULL
cast(d AS string) e.g. +262143-01-01 query fails: Cast error: Failed to convert 95026237 to temporal for Date32
last_day(d) a date query fails: Spark last_day: Unable to convert days value 108853388 to date

dayofweek, add_months, date_add, unix_date and date comparisons already match Spark for these values.

Expected behavior

These expressions should return the same values as Spark for every date that Spark DateType can hold, or fall back to Spark when the native kernel cannot represent the value.

Additional context

Casting such dates to TIMESTAMP panics, which is tracked separately in #5456. date_format is not listed because Spark itself throws ArithmeticException: long overflow for year 300000.

Activity

  1. andygrove commented on Oct 10, 2026

    @andygrove
    Member

    We should move to jif. I created the first arrow-rs PR towards this: apache/arrow-rs#11482

  2. added
    bugSomething isn't working
    priority:mediumFunctional bugs, performance regressions, broken features
    and removed on Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:expressionsExpression evaluationbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions