Skip to content

perf: Drop RowConverter from GroupOrderingPartial - #14566

Merged
2010YOUY01 merged 8 commits into
apache:mainfrom
ctsk:faster-partial-groupordering
Feb 17, 2025
Merged

2010YOUY01 merged 8 commits into
apache:mainfrom
ctsk:faster-partial-groupordering

Conversation

@ctsk

@ctsk ctsk commented Feb 9, 2025

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Faster is better?

What changes are included in this PR?

Are these changes tested?

Yes, added a new test.

Are there any user-facing changes?

No.

(The input_schema argument of GroupOrderingPartial::try_new is no longer used and can be removed, this is not included in this PR)

@github-actions github-actions Bot added the physical-expr Changes to the physical-expr crates label Feb 9, 2025
@ctsk
ctsk force-pushed the faster-partial-groupordering branch from d7e7376 to a7324c7 Compare February 10, 2025 08:18
@ctsk

ctsk commented Feb 10, 2025

Copy link
Copy Markdown
Contributor Author

Micro-benchmark results

  • group_ordering_$n benchmarks the partial group ordering where the ordering contains $n columns. Each Columns consists of 8192 Int32s.
group_ordering_partial/order_indices_1
                        time:   [16.814 µs 17.340 µs 17.934 µs]
                        change: [-71.063% -70.318% -69.568%] (p = 0.00 < 0.05)
                        Performance has improved.
group_ordering_partial/order_indices_2
                        time:   [23.585 µs 23.907 µs 24.176 µs]
                        change: [-65.486% -64.711% -63.930%] (p = 0.00 < 0.05)
                        Performance has improved.
group_ordering_partial/order_indices_4
                        time:   [28.452 µs 28.603 µs 28.794 µs]
                        change: [-52.826% -52.404% -51.942%] (p = 0.00 < 0.05)
                        Performance has improved.
group_ordering_partial/order_indices_8
                        time:   [34.737 µs 34.828 µs 34.935 µs]
                        change: [-64.151% -63.876% -63.589%] (p = 0.00 < 0.05)
                        Performance has improved.

@ctsk
ctsk force-pushed the faster-partial-groupordering branch from cccc8ea to 0d778f5 Compare February 10, 2025 08:40
@ctsk ctsk changed the title Drop RowConverter from GroupOrderingPartial perf: Drop RowConverter from GroupOrderingPartial Feb 10, 2025
@ctsk
ctsk force-pushed the faster-partial-groupordering branch from 0d778f5 to ec21860 Compare February 12, 2025 16:04
@2010YOUY01

Copy link
Copy Markdown
Contributor

Thank you, this looks good to me. Let's get the CI fixed.

https://github.com/apache/datafusion/blob/main/datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs should have good coverage for edge cases in group by with partial ordering, so I think this change is safe

@ctsk
ctsk force-pushed the faster-partial-groupordering branch from ec21860 to 2961710 Compare February 13, 2025 11:11
@ctsk

ctsk commented Feb 13, 2025

Copy link
Copy Markdown
Contributor Author

Thank you for taking the time to review!

impl GroupOrderingPartial {
pub fn try_new(
input_schema: &Schema,
_input_schema: &Schema,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not remove this arg?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way we can avoid API change, though I don't have a strong preference

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can add a comment or something as a follow on pr

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, the input_schema argument in order::GroupOrdering::try_new will also be unnecessary

impl GroupOrdering {
/// Create a `GroupOrdering` for the specified ordering
pub fn try_new(
input_schema: &Schema,
mode: &InputOrderMode,
ordering: &LexOrdering,
) -> Result<Self> {
match mode {
InputOrderMode::Linear => Ok(GroupOrdering::None),
InputOrderMode::PartiallySorted(order_indices) => {
GroupOrderingPartial::try_new(input_schema, order_indices, ordering)
.map(GroupOrdering::Partial)
}
InputOrderMode::Sorted => Ok(GroupOrdering::Full(GroupOrderingFull::new())),
}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a TODO comment to the code

Comment thread datafusion/physical-plan/src/aggregates/order/partial.rs Outdated
@2010YOUY01
2010YOUY01 merged commit 2fce1ef into apache:main Feb 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve GroupOrderingPartial performance

5 participants