Skip to content

Make the Delta contrib's file-planning boundary explicit #5946

Open

Description

@dwsmith1983

What is the problem the feature request solves?

planningHelper in CometDeltaNativeScanExec.scala reuses core's listing, splitting and dynamic partition pruning, but its contract is implicit: partition filters are supplied separately while data filters come from originalPlan, and predicates used for file selection are not distinguished from predicates the reader evaluates.

Describe the potential solution

A contract that takes the original scan plus the current partition and data filters and returns the final FilePartition objects with their metadata, with the existing reader-free helper as the implementation and the serializer consuming only its result. Plan copies and filter rewrites keep the current protection against stale cached selection. Regression tests compare serialized paths, order, split starts and lengths, partition values and metadata against Spark's final partitions, including a non-DV file with a nonzero split start and dynamic partition pruning with AQE on and off.

Additional context

Raised in the #5365 review (item 4); filed as a follow-up.

Activity

  1. dwsmith1983 commented on Oct 9, 2026

    @dwsmith1983
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions