Skip to content

Use arrow row format in GroupByHash rather than datafusion row format #4179

Open

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Arrow now has a row format https://docs.rs/arrow/26.0.0/arrow/row/index.html that is both well documented and likely to get additional functionality (as it will be used for sorting, etc)

It would be good to use that in DataFusion to benefit from upstream efforts including more type support as it is added

Describe the solution you'd like
It would be great to use the Arrow row format instead of Compact layout: https://github.com/apache/arrow-datafusion/blob/f2f846512ab032845de5dcee768a8a69ddf17eac/datafusion/row/src/layout.rs#L68

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
As part of #3940 and #3941 I believe @crepererum is working on this feature

cc @yjshen and @tustvold

Activity

  1. tustvold commented on Jan 17, 2023

    @tustvold
    Contributor

    I believe this was closed by #4830

  2. crepererum commented on Jan 17, 2023

    @crepererum
    Contributor

    Only for the group key, not for the group state.

  3. alamb commented on Jan 17, 2023

    @alamb
    ContributorAuthor

    I think we should probably file a follow on ticket for the group state (aka the aggregates) as I am not sure the existing row format is actually the right thing. I'll try and sort this out tomorrow when filing tickets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions