Native Parquet writer ignores the datetime rebase modes, so Spark rejects or shifts its pre-1582 dates #7047

Describe the bug

The native Parquet writer ignores spark.sql.parquet.datetimeRebaseModeInWrite, and it writes none of the footer keys that Spark's readers use to decide whether a file needs rebasing. That affects dates and timestamps before 1582-10-15, at the top level and inside structs, arrays and maps, in three ways:

• Spark 3.4 and 3.5 refuse the write, but the native writer doesn't. In the EXCEPTION mode those versions default to, Spark fails with [INCONSISTENT_BEHAVIOR_CROSS_VERSION.WRITE_ANCIENT_DATETIME]. The native writer writes the file.
• Spark 3.4 and 3.5 then can't read the file back. The file has no org.apache.spark.version key, so DataSourceUtils.datetimeRebaseSpec falls back to spark.sql.parquet.datetimeRebaseModeInRead, also EXCEPTION by default, and the read fails with [INCONSISTENT_BEHAVIOR_CROSS_VERSION.READ_ANCIENT_DATETIME]. A file Spark writes in CORRECTED mode carries the key and reads back fine.
• LEGACY read mode shifts the values silently, on every version. Spark treats the native file's proleptic Gregorian values as hybrid-calendar values and rebases them, so 1000-01-01 reads back as 0999-12-27. In the other direction, with datetimeRebaseModeInWrite=LEGACY Spark rebases on write and records org.apache.spark.legacyDateTime, while the native writer writes proleptic values and no key. Readers that expect the hybrid calendar, such as Spark 2.4 and Hive 2.x, then see shifted dates.

ParquetWriterExec sets only the compression on its WriterProperties (native/core/src/execution/operators/parquet_writer.rs), and nothing on the JVM side reads the rebase configs for a native write. Comet's own scan reads the native files back correctly in every mode, because it never rebases (#5010).

Steps to reproduce

Write the source with Comet disabled, so it holds proleptic values:

spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.parquet.datetimeRebaseModeInWrite", "CORRECTED")
spark.sql("""SELECT date'1000-01-01' AS d, named_struct('d', date'1200-03-01') AS s,
  array(date'1400-06-01') AS a""").write.parquet("/tmp/src")
spark.conf.unset("spark.sql.parquet.datetimeRebaseModeInWrite")

Then write it natively and read the result with Comet disabled:

spark.conf.set("spark.comet.write.parquet.enabled", "true")
// Spark 4.x. On Spark 3.x the key is spark.comet.operator.DataWritingCommandExec.allowIncompatible
spark.conf.set("spark.comet.operator.WriteFilesExec.allowIncompatible", "true")
spark.read.parquet("/tmp/src").write.parquet("/tmp/native")

spark.conf.set("spark.comet.enabled", "false")
spark.read.parquet("/tmp/native").show()
spark.conf.set("spark.sql.parquet.datetimeRebaseModeInRead", "LEGACY")
spark.read.parquet("/tmp/native").show()

On main at c230bf4:

Timestamp columns shift the same way.

Expected behavior

The native writer does what Spark's writer does for the effective write mode, or falls back:

• CORRECTED: write org.apache.spark.version into the footer ([COMET NATIVE WRITER] Write Spark version into Parquet metadata #3427), so Spark readers treat the file as a Spark 3+ file that needs no rebasing.
• EXCEPTION: fail on an ancient value the way Spark does, or fall back for writes with date or timestamp columns.
• LEGACY: fall back, or rebase and write org.apache.spark.legacyDateTime and org.apache.spark.timeZone as Spark does.

Additional context

Found in a nested-type audit of the native writers. #4180 lists datetimeRebaseModeInWrite and int96RebaseModeInWrite among the configs to audit, and #5010 tracks the read side. The native writer never writes INT96, so int96RebaseModeInWrite has no effect on it today.