Skip to content

Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning #5553

Open

Description

@peterxcli

Context

#5177 makes the native Parquet reader's TIMESTAMP_MILLIS -> microseconds conversion checked for top-level columns, matching Spark's millisToMicros (Math.multiplyExact). Two cases intentionally keep the pre-existing safe-cast behavior (overflow -> NULL) instead of erroring like Spark:

  1. Nested fields (struct children, list/map elements): the conversion inside parquet_convert_struct_to_struct / list / map recursion is unchecked.
  2. Scans whose data filters reference nested fields (GetStructField): the whole scan falls back to the safe cast (SparkParquetOptions.checked_timestamp_overflow = false, set in init_datasource_exec).

Why

Spark only avoids the overflow error for filtered-out values through row-group statistics pruning (ParquetFilters supports nested column predicates). DataFusion currently can neither:

  • prune nested-field predicates — datafusion-pruning says "PruningPredicate does not support pruning on nested fields yet" (pruning_predicate.rs), nor
  • evaluate them as Parquet row filters — can_expr_be_pushed_down_with_schemas in datafusion-datasource-parquet/src/row_filter.rs classifies struct columns as non-pushable.

So a checked conversion on these paths would decode row groups Spark prunes and fail queries Spark answers with zero rows (see the review discussion on #5177: ts IN (...), ts <=> ..., and s.ts < ... reproducers).

Remaining divergence vs Spark

  • A nested-predicate scan that Spark fails to prune (mixed row-group statistics) throws ArithmeticException in Spark but returns NULL for overflowing top-level values in Comet.
  • A direct read of an overflowing nested TIMESTAMP_MILLIS field throws in Spark but returns NULL in Comet (pre-existing behavior on main).

Proposed work

Once DataFusion supports nested-field pruning (and ideally nested row filters), remove the checked_timestamp_overflow fallback and extend the checked conversion to nested fields, plus extend the millisecond-domain predicate rewrite in SparkPhysicalExprAdapter to GetStructField-wrapped conversions.

Regression coverage lives in ParquetReadSuite:

  • "TIMESTAMP_MILLIS overflow fails in native scan"
  • "TIMESTAMP_MILLIS overflow rows skipped by filter pruning do not fail"

Related: #5517 (error fidelity for the overflow exception).

Activity

  1. peterxcli commented on Aug 29, 2026

    @peterxcli
    MemberAuthor

    take

  2. added
    bugSomething isn't working
    priority:criticalData corruption, silent wrong results, security issues
    area:scanParquet scan / data reading
    and removed on Aug 31, 2026
  3. added this to the 1.1.0 milestone on Sep 11, 2026
  4. modified the milestones: 1.1.0, 1.2.0 on Sep 29, 2026
  5. dwsmith1983 commented on Oct 4, 2026

    @dwsmith1983
    Contributor

    #5740 fixed nested overflow checking in unfiltered scans. The filtered-scan part, which depends on apache/datafusion#20871, is tracked in #5739, so I think this one can be closed in favor of it.

  6. modified the milestones: 1.2.0, 1.3.0 on Oct 15, 2026
  7. kazuyukitanimura commented on Oct 15, 2026

    @kazuyukitanimura
    Contributor

    Hi @peterxcli
    Do you think we can close this now based on @dwsmith1983 's comment?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingbugSomething isn't workingcorrectnesspriority:criticalData corruption, silent wrong results, security issues

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions