Native timezone conversions stop applying DST after 2099, while Spark keeps applying the zone's rules #6816

What / Why

Comet converts between instants and local time with chrono-tz, which stores each zone's precomputed transitions up to its last one, around 2099 for zones that observe DST. After that it applies the zone's standard offset all year. Spark uses java.time, whose ZoneRules keep applying the zone's last transition rules forever. So after about 2100, a native expression that works in local time in a DST zone can be off by the DST offset for the summer months.

For example, in an America/Los_Angeles session with spark.comet.expression.TruncTimestamp.allowIncompatible=true, date_trunc('DAY', TIMESTAMP'2300-07-01 12:00:00') is 2300-07-01T07:00:00Z (PDT) in Spark and 2300-07-01T08:00:00Z (PST) in Comet. This happens both for literal formats and for formats that come from a column. The same offset applies to hour and to casts from timestamps to strings and dates, which run natively by default.

The compatibility guide already documents the difference ("approximately beyond year 2100"). This issue tracks fixing it rather than documenting it.

Possible fixes

#4754 evaluates moving from chrono to Jiff, which applies a zone's POSIX TZ rule after its last transition, as java.time does.

Found during review of #6354.