Skip to content

Optimize hash operations and byte-wise row comparisons for dictionary encoding #21466

Closed

Description

@Rich-T-kid

Two related performance improvements for dictionary-encoded columns during aggregation:

Byte-wise row comparisons : when checking for existing groups, group_rows.row(row) == group_values.row(*group_idx) compares raw bytes. Since dictionary encoding represents repeated values as integer indices, comparing those integers directly is significantly cheaper.
Hash operations : hash calls on dictionary-encoded columns currently operate on the full string/primitive values. Replacing these with hashes of the integer dictionary keys is a minor but meaningful performance gain.

Relevant Packages

Hash operations: datafusion-common
Byte-wise row comparisons: datafusion-physical-plan

Goals

  1. Byte-wise row comparisons updated to compare dictionary integer indices instead of raw bytes
  2. Hash calls on dictionary-encoded columns updated to use integer key values rather than decoded primitives
  3. Performance benchmarks demonstrate measurable improvement
  4. Regression tests for hashing confirm dictionary encoding changes don't violate hash guarantees

Note: Dictionary key→value mappings are not stable across RecordBatches;The same key index can map to a different value in a different batch. Any new logic must not assume otherwise.

Activity

  1. Rich-T-kid commented on Apr 8, 2026

    @Rich-T-kid
    ContributorAuthor

    this falls under #7000 & performance and regression test that will be included in the PR fall under #8791.

  2. Rich-T-kid commented on Apr 8, 2026

    @Rich-T-kid
    ContributorAuthor

    take

  3. Rich-T-kid commented on Aug 5, 2026

    @Rich-T-kid
    ContributorAuthor

    #23187 will close this when its merged

  4. Rich-T-kid commented on Aug 16, 2026

    @Rich-T-kid
    ContributorAuthor

    #23187 closes this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions