You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning #5553
#5177 makes the native Parquet reader's TIMESTAMP_MILLIS -> microseconds conversion checked for top-level columns, matching Spark's millisToMicros (Math.multiplyExact). Two cases intentionally keep the pre-existing safe-cast behavior (overflow -> NULL) instead of erroring like Spark:
Nested fields (struct children, list/map elements): the conversion inside parquet_convert_struct_to_struct / list / map recursion is unchecked.
Scans whose data filters reference nested fields (GetStructField): the whole scan falls back to the safe cast (SparkParquetOptions.checked_timestamp_overflow = false, set in init_datasource_exec).
Why
Spark only avoids the overflow error for filtered-out values through row-group statistics pruning (ParquetFilters supports nested column predicates). DataFusion currently can neither:
prune nested-field predicates — datafusion-pruning says "PruningPredicate does not support pruning on nested fields yet" (pruning_predicate.rs), nor
evaluate them as Parquet row filters — can_expr_be_pushed_down_with_schemas in datafusion-datasource-parquet/src/row_filter.rs classifies struct columns as non-pushable.
So a checked conversion on these paths would decode row groups Spark prunes and fail queries Spark answers with zero rows (see the review discussion on #5177: ts IN (...), ts <=> ..., and s.ts < ... reproducers).
Remaining divergence vs Spark
A nested-predicate scan that Spark fails to prune (mixed row-group statistics) throws ArithmeticException in Spark but returns NULL for overflowing top-level values in Comet.
A direct read of an overflowing nested TIMESTAMP_MILLIS field throws in Spark but returns NULL in Comet (pre-existing behavior on main).
Proposed work
Once DataFusion supports nested-field pruning (and ideally nested row filters), remove the checked_timestamp_overflow fallback and extend the checked conversion to nested fields, plus extend the millisecond-domain predicate rewrite in SparkPhysicalExprAdapter to GetStructField-wrapped conversions.
Regression coverage lives in ParquetReadSuite:
"TIMESTAMP_MILLIS overflow fails in native scan"
"TIMESTAMP_MILLIS overflow rows skipped by filter pruning do not fail"
Related: #5517 (error fidelity for the overflow exception).
#5740 fixed nested overflow checking in unfiltered scans. The filtered-scan part, which depends on apache/datafusion#20871, is tracked in #5739, so I think this one can be closed in favor of it.
Activity
peterxcli commentedon Aug 29, 2026
take
dwsmith1983 commentedon Oct 4, 2026
#5740 fixed nested overflow checking in unfiltered scans. The filtered-scan part, which depends on apache/datafusion#20871, is tracked in #5739, so I think this one can be closed in favor of it.
kazuyukitanimura commentedon Oct 15, 2026
Hi @peterxcli
Do you think we can close this now based on @dwsmith1983 's comment?