Skip to content

Delegate Delta physical schema preparation to DeltaParquetFileFormat.prepareSchemaForRead #5949

Open

Description

@dwsmith1983

What is the problem the feature request solves?

toPhysical in the Delta contrib duplicates createPhysicalSchema and the name-mode field-id stripping that DeltaParquetFileFormat.prepareSchemaForRead already implements in every supported Delta version.

Describe the potential solution

Delegate to prepareSchemaForRead with the generated bookkeeping columns excluded, and keep a clear division between the physical read schemas and the logical output types in common.fields so native output adaptation can restore logical nested types before parent expressions run, with mapped complex predicates staying residual until they bind safely to the physical schema. The current ID-mode and mapped-nested fallbacks stay until that adaptation exists. Useful controls: upgrade and rename, to_json of mapped structs, missing nested children, and version-specific null-struct behavior; #5661 documents the support boundary.

Additional context

Raised in the #5365 review (item 9); filed as a follow-up.

Activity

  1. dwsmith1983 commented on Oct 9, 2026

    @dwsmith1983
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions