Skip to content

Parquet reader: Add support for reading row index metadata columns #3432

Open
Open
Parquet reader: Add support for reading row index metadata columns#3432
Assignees
mbutrovich
Labels
enhancementNew feature or requestnative_datafusionSpecific to native_datafusion scan type
Milestone
1.3.0

Description

@andygrove
andygrove
opened on Feb 6, 2026

What is the problem the feature request solves?

The native_datafusion scan currently falls back to Spark for queries that read row index metadata columns. It would be nice to support them.

Describe the potential solution

No response

Additional context

No response

Activity

  1. andygrove
    added
    enhancementNew feature or request
    on Feb 6, 2026
  2. andygrove
    added a commit that references this issue on Mar 31, 2026

    fix: add issue link for row index generation skips in Spark SQL test …

    b4a5aa7
  3. mbutrovich
    self-assigned this
    on May 4, 2026

  4. mbutrovich commented on May 4, 2026

    @mbutrovich
    mbutrovich
    on May 4, 2026
    Contributor

    Relevant: apache/arrow-rs#8641 apache/arrow-rs#8715

  5. mbutrovich commented on May 5, 2026

    @mbutrovich
    mbutrovich
    on May 5, 2026
    Contributor

    Opened upstream DataFusion PR that we'll hopefully pick up in 54.0:

    apache/datafusion#22026

  6. andygrove
    added this to the 0.17.0 milestone on May 13, 2026
  7. andygrove
    added this to Comet Developmenton May 13, 2026
  8. github-project-automation
    moved this to Todo in Comet Developmenton May 13, 2026
  9. mbutrovich
    moved this from Todo to In progress in Comet Developmenton May 13, 2026
  10. andygrove
    added
    native_datafusionSpecific to native_datafusion scan type
    on May 16, 2026
  11. pull
    added a commit that references this issue on May 28, 2026

    feat: Plumb Parquet virtual columns (row_number) through TableSchema …

    2888bce
  12. kazuyukitanimura
    modified the milestones: 0.17.0, 1.0.0 on Jun 25, 2026

  13. mbutrovich commented on Jul 22, 2026

    @mbutrovich
    mbutrovich
    on Jul 22, 2026
    Contributor

    Note that this won't land until DF 55.0.

  14. mbutrovich
    modified the milestones: 1.0.0, 1.1.0 on Jul 22, 2026
  15. mbutrovich
    changed the title [-][native_datafusion] Add support for reading row index metadata columns[/-] [+]Parquet reader: Add support for reading row index metadata columns[/+] on Jul 22, 2026
  16. andygrove
    modified the milestones: 1.1.0, 1.2.0 on Sep 11, 2026
  17. andygrove
    modified the milestones: 1.2.0, 1.3.0 on Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestnative_datafusionSpecific to native_datafusion scan type

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions