Skip to content

Document Schema metadata expectations #12736

Open
Open
Document Schema metadata expectations#12736
Assignees
alamb
Labels
enhancementNew feature or request

Description

@alamb
alamb
opened on Oct 3, 2024

Is your feature request related to a problem or challenge?

There is an (implicit) assumption that metadata attached to Schema is preserved during certain operations in DataFusion.

However, this expectation is clearly not well tested or documented (e.g. see #12733)

Describe the solution you'd like

I would like the assumptions documented

Describe alternatives you've considered

I suggest documentation on in https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html that explains the high level assumptions

Then add a note /link to that section from the optimizers:
https://docs.rs/datafusion/latest/datafusion/optimizer/trait.AnalyzerRule.html
https://docs.rs/datafusion/latest/datafusion/optimizer/trait.OptimizerRule.html
https://docs.rs/datafusion/latest/datafusion/physical_optimizer/trait.PhysicalOptimizerRule.html

My understanding of the high level assumptions are:

  • schema level metadata: always passed through
  • field level metadata: when there is a clear 1-1 correspondence from an input column with metadata to an output column, the metadata should be preserved

Examples

  • PROJECT(a, b+c) --> field metadata ona should be preserved, no field metadata on b+c
  • SUM(a) .. GROUP BY b --> field metadata on b is preserved, not on a

Additional context

No response

Activity

  1. alamb
    added
    enhancementNew feature or request
    on Oct 3, 2024

  2. alamb commented on Oct 3, 2024

    @alamb
    alamb
    on Oct 3, 2024
    ContributorAuthor

    I believe @wiedld plans to work on this

  3. wiedld commented on Oct 3, 2024

    @wiedld
    wiedld
    on Oct 3, 2024
    Contributor

    take

  4. github-actions
    assigned
    wiedld
    on Oct 3, 2024

  5. alamb commented on Oct 21, 2024

    @alamb
    alamb
    on Oct 21, 2024
    ContributorAuthor

    I will take a shot at documenting this

  6. alamb
    assigned
    alamb
    and unassigned
    wiedld
    on Oct 21, 2024

  7. alamb commented on Nov 13, 2024

    @alamb
    alamb
    on Nov 13, 2024
    ContributorAuthor

    #13305 (comment) has some additional context

  8. westonpace commented on Jan 24, 2025

    @westonpace
    westonpace
    on Jan 24, 2025
    Member

    I'm not sure if this is related or not but I encountered an error during optimization:

    Error: join_selection caused by Internal error: PhysicalOptimizer rule 'join_selection' failed, due to generate a different schema, original schema: Schema { fields: [Field { name: "a", ...  }], metadata: {metadata_from_table_one}, new schema: Schema { fields: [Field { name: "a", ... }], metadata: {metadata_from_table_two}. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

    I think the join operator is either returning the metadata from the first or second input and so an optimization that swaps the join order results in the output schema of the join operator changing and this causes the optimizer to bail.

  9. alamb commented on Jan 24, 2025

    @alamb
    alamb
    on Jan 24, 2025
    ContributorAuthor

    That definitely sounds like a bug

  10. wiedld commented on Aug 18, 2025

    @wiedld
    wiedld
    on Aug 18, 2025
    Contributor

    @alamb -- I'm taking a look back at this ticket. There has been some changes, and I'm unclear as to long term plans.

    There is a new FieldMetadata which only applies to literals and unifies it's handling? What about non-literal fields?

    How (or is) this related to a consolidation of how schema field metadata is handled? Are we unifying a single wrapper abstraction to handle how merging (e.g. union), precedence (e.g. joins?), aggregates, lists, and/or casting is handled?

  11. alamb commented on Aug 18, 2025

    @alamb
    alamb
    on Aug 18, 2025
    ContributorAuthor

    @alamb -- I'm taking a look back at this ticket. There has been some changes, and I'm unclear as to long term plans.

    There is a new FieldMetadata which only applies to literals and unifies it's handling? What about non-literal fields?

    How (or is) this related to a consolidation of how schema field metadata is handled? Are we unifying a single wrapper abstraction to handle how merging (e.g. union), precedence (e.g. joins?), aggregates, lists, and/or casting is handled?

    TLDR is I don't know of any unified plan.

    What I would recommend doing is starting by documenting what (little) is known today (about projections and aliases)

    Then as we evolve and the plans become clearer, we will at least have a place to improve the documenetion

    I think starting with something even if it is incomplete, is better than the status quo (nothing)

  12. wiedld commented on Aug 19, 2025

    @wiedld
    wiedld
    on Aug 19, 2025
    Contributor

    Non-exhaustive list, as I find things:

  13. adriangb commented on Dec 15, 2025

    @adriangb
    adriangb
    on Dec 15, 2025
    Contributor

    I found a bug with first_value / last_value: #19335

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions