Repository navigation
Support zero copy hash repartitioning for Hash Join #15382
Description
Is your feature request related to a problem or challenge?
Currently RepartitionExec: partitioning=Hash will be added whenever a hash
The benefit is increased parallelism, but at the cost of copying the entire table (in a not-so efficient way).
We should consider lowering the cost of repartitioning by not having to copy the input.
Describe the solution you'd like
Instead of repartitioning the right side and left side input in RepartitionExec, support repartitioning the inputs based on a selection vector.
Instead of taking the RecordBatch, we can consider doing the following:
- Add a (boolean) selection vector as output column for each output partition. I.e.
truemeans the row is selected for the partition. - The rest of the
RecordBatchremains unchanged (i.e. no copy). - CoalesceBatchesExec is no longer needed for the output (reducing another copy)
- In the hash join algorith,, only try matching input indices for the selection vector for each partition. This seems not that hard as
get_matched_indicesalready supports getting an input of input indices + hashes.
Dependencies
Describe alternatives you've considered
The partitioning could be done inside the hash join algorithm, however this would add more complexity to each operator (join / aggregates) to do hash-repartitioning inside
Additional context
No response
Activity
[-]Support zero copy hash repartitioning inside Hash Join[/-][+]Support zero copy hash repartitioning for Hash Join[/+]zebsme commentedon Mar 24, 2025
take
alamb commentedon Mar 24, 2025
@zebsme this is likely a fairly tricky thing to implement, just FYI
zebsme commentedon Mar 25, 2025
If anyone's interested in this, feel free to try :)
Dandandan commentedon Mar 25, 2025
Yes, I agree with @alamb I don't consider this super easy. However, feel free to try and implement some steps and let us know when you need help @zebsme .
Probably it makes sense to split this issue in some subtasks, these are the ones I can think of and I think the individually they might be doable for an "advanced beginner":
RecordBatch) inRepartitionExec. The array outputstruefor each row that hashash % partition == 0(and false if not).We probably want to think about some general way of representing / using selection vectors in
RecordBatch(or as wrappedRecordBatch) as well but this requires some more design / discussion.goldmedal commentedon Mar 25, 2025
I'll work on #15383, but I guess there are some duplicate works here.
zebsme commentedon Mar 25, 2025
That makes sense. Perhaps we should address #15420 first - I'd be happy to help review the code.
Rich-T-kid commentedon Sep 5, 2026
@Dandandan Wouldn't this mean our memory consumption would grow? In this model, we'd keep each record batch alive instead of slicing it apart and allocating a new one. If record batches have backing arrays/buffers that remain referenced, won't they stick around for the duration of the query?
besides that I think this is a pretty cool idea
Dandandan commentedon Sep 5, 2026
I think it depends - if the join reduces data, then it could grow and we maybe would need to compact it / detect this case.
In other cases (each row has at least one match), not copying it might actually reduce peak memory.