fix: cast string to date for years outside chrono's NaiveDate range

[viirya]

Which issue does this PR close?

Closes #6842.

Depends on #5457, which makes the downstream date-to-timestamp cast safe for these dates (#5456). This PR should merge after it.

Rationale for this change

Spark's stringToDate accepts a year of up to 7 digits (optionally signed) and only rejects a date when LocalDate.of fails or the epoch day overflows Int, which happens past 5881580-07-11 or before -5877641-06-23. Comet's date_parser computed the same epoch day without chrono, but then returned NULL in every eval mode for any year outside chrono's NaiveDate range (-262143 to 262142). As a result, CAST, TRY_CAST and to_date returned NULL for valid dates such as 294248-01-01 or 1000000-01-01, even in ANSI mode.

What changes are included in this PR?

• Remove the chrono year bound in date_parser. Its remaining checks match Spark: an invalid calendar date or an epoch day that overflows i32 gives NULL in LEGACY/TRY mode and CAST_INVALID_INPUT in ANSI mode.
• Remove the year-range limitation from the cast compatibility guide.

Known downstream limitations

This makes CAST(string AS DATE) return dates outside chrono's NaiveDate range, which previously came back as NULL. Some native consumers already mishandle such dates when they come from Parquet or make_date, and this change lets them come from strings too: casting them to TIMESTAMP panics (#5456), and several date-field functions and CAST(date AS STRING) return NULL or fail (#6843). For those queries, a wrong NULL from the string cast becomes an error from the downstream expression.

How are these changes tested?

• Rust unit tests in string.rs now assert exact epoch days for years past chrono's range, including leap days, a + sign and a T suffix, and both i32 boundaries. They also check that one day past either boundary, invalid calendar dates past chrono's range, and 8-digit years are rejected.
• New SQL file test expressions/cast/cast_string_to_date_wide_year.sql, run with ANSI enabled and disabled. It checks unix_date of cast, try_cast and to_date, plus the raw date values, and fails on main. The raw-date query leaves out the two Int-boundary rows because Spark cannot convert them to java.sql.Date when collecting.
• CometNativeCastSuite "cast StringType to DateType" adds explicit wide-year inputs. Its fuzz filter now drops only 7-digit years near the Int limit, for the same java.sql.Date reason.

[sunchao]

Reviewed the entire four-file diff from bfacc43a50aba13da4fa8a828db52309541191b4 to 8d0f144efe72459b6525903f3da4f3c92dd2353b. The PR is non-draft. Used review-comet-pr and review-comet-expression-pr. The snapshot and live discussion contained no reviews or comments.

Summary

• Prior state and problem: CAST(string AS DATE), TRY_CAST and unformatted to_date returned null for valid years outside chrono’s range, although Spark accepts dates throughout the signed 32-bit epoch-day range.
• Design approach: Remove the chrono year guard and retain Gregorian calendar validation plus checked conversion to i32.
• Correctness: The parser’s new boundary cases pass. However, newly accepted dates reach an unchanged timestamp consumer that panics. This materially worsens the acknowledged concern in #5456 and is the P2 finding below.
• Compatibility analysis: Checked Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3 and 4.2.0 sources. Their date parsers agree on seven-digit years, calendar validation and epoch-day overflow. Spark’s unformatted to_date delegates to a date cast. The latest release branch, branch-1.1, has the same parser and range guard as the base.
• Key design decisions: Preserve the existing Legacy/Try null versus ANSI error behavior for malformed input. Exercise extreme dates through unix_date to avoid Spark’s separate java.sql.Date collection limitation.
• Implementation sketch: Update resolve_epoch_day, replace out-of-range null assertions with exact epoch-day assertions, add SQL tests under both ANSI settings, widen Scala fuzz coverage and update the compatibility template.
• Performance: The production change removes a range check without introducing allocations or additional passes. No performance regression was identified from the implementation. No benchmark was run or speedup established.
• Design: The parser change is small and straightforward. Expanding its output domain requires safe downstream consumption, which the current timestamp path does not provide.
• Abstraction & complexity: Reusing ymd_to_epoch_day avoids another calendar implementation. Separate valid-input and malformed-input assertion helpers remain simple and useful.
• Behavioral changes worth calling out: Standalone wide-year casts now return Spark-compatible dates. Composed date-to-timestamp casts can change from a null result to a query failure, including in Try mode.
• Suggested improvements: Make the affected timestamp consumer handle wide dates correctly, or route affected compositions through Spark, before removing this protection. Add a column-based composition regression covering all evaluation modes.

Validation: date_parser_test passed. A disposable native integration test confirmed the timestamp panic in Legacy, ANSI and Try modes, checked the exact base parser’s null result, and verified that the corresponding null input avoids the panic. Spark 3.5.9 date utilities independently returned the expected epoch day and UTC microseconds. The disposable test was removed and the worktree is clean. Full JVM/SQL suites and benchmarks were not run locally.

Exact-head CI: At the final inspection, 23 checks succeeded, 3 remained in progress and 14 were skipped. Native build, Rust tests, scans, shuffle and TPC-H passed. Expressions, execution and TPC-DS remained running. Spark SQL compatibility jobs and macOS were skipped. No failed checks were reported. Run: https://github.com/apache/datafusion-comet/actions/runs/38028364407

[sunchao]

[P2] Handle wide dates in timestamp consumers before removing this guard. With a string column containing 262143-01-01 and the session timezone set to UTC, CAST(CAST(s AS DATE) AS TIMESTAMP) now passes epoch day 95026237 into cast_date_to_timestamp, whose chrono addition panics at temporal.rs:74. Spark successfully returns 8210266876800000000 microseconds. The base returned null from the inner cast and avoided the panic, so this PR materially expands #5456 to previously non-failing string-input queries. One such row can fail a mixed batch, even with Try evaluation. Please make the consumer support these dates or fall back to Spark for affected compositions, and cover the composition in regression tests.

Evidence: A temporary integration test invoked the exact-head spark_cast on ['2020-01-01', NULL, '262143-01-01'], then cast the resulting Date32 array to Timestamp(Microsecond, UTC). Legacy, ANSI and Try each panicked with 'NaiveDate + TimeDelta overflowed' at conversion_funcs/temporal.rs:74. The exact base parser returned None for the wide input in all three modes, and the corresponding timestamp conversion preserved null without panicking. Spark 3.5.9 stringToDate returned Some(95026237), and daysToMicros with UTC returned 8210266876800000000. SQL trigger with Comet enabled: SET spark.sql.session.timeZone=UTC; CREATE TABLE wide_date_review(s STRING) USING parquet; INSERT INTO wide_date_review VALUES ('2020-01-01'), ('262143-01-01'); SELECT unix_micros(CAST(CAST(s AS DATE) AS TIMESTAMP)) FROM wide_date_review. Validation executed the native composition and Spark utility calls, not the full SQL query.

[viirya]

Agreed that this composition needs to be safe before this lands. The panic itself is #5456: such dates already reach cast_date_to_timestamp from Parquet date columns and from native make_date, and this PR adds string input as another source. Since #5457 fixes that consumer, I'd like to keep this PR scoped to the parser and merge it after #5457. Once #5457 is in, I'll rebase and add a column-based regression for CAST(CAST(s AS DATE) AS TIMESTAMP) and TIMESTAMP_NTZ under LEGACY, ANSI and TRY.

One note for that test: in the SQL trigger above, unix_micros runs through the codegen dispatcher, so the inner casts are evaluated by Spark and never reach the native kernel. The regression will use a consumer that stays native, such as CAST(... AS BIGINT) on the timestamp.

[andygrove]

WE should just move to gif jiff instead of chrono, IMO. Work on this has started in Arrow: apache/arrow-rs#11482

edit: I meant https://crates.io/crates/jiff, not gif.

[viirya]

+1 on jiff for timezone handling. apache/arrow-rs#11482 fixes a real class of bugs here: chrono-tz stops applying DST rules after its precomputed table, while jiff follows the TZif footer the way java.time does, which is what #6816 needs.

The year range in this PR is a separate limit, though, and jiff doesn't widen it. jiff's civil::Date only covers years -9999..=9999, which is narrower than chrono's ±262142, while Spark DateType reaches about ±5.88 million years. So any kernel that converts a Spark date into a library calendar type will hit that limit, whichever library it uses. This PR avoids it because date_parser computes the epoch day with integer arithmetic (ymd_to_epoch_day) and never goes through chrono, the same approach #5443 took for make_date. The downstream consumers (#5456, #6843) have the same two options under either library: compute in integer epoch days, or fall back to Spark for values the library can't represent.