Repository navigation
Implement Radix Hash Join #18939
Description
Is your feature request related to a problem or challenge?
Hash Joins with large build side & lots of hash-duplicates are relatively slow in DataFusion.
The cost seems largely associated with traversing the chain of duplicates (chain_traverse) (1) + which is known to be very cache-inefficient, as the access pattern is mostly random.
Currently, we implement hash joins partitioned by hash, but we can implement a more efficient algorithm (radix hash join) that splits build data into smaller tables that individually mostly fits in CPU caches and allow more efficient access patterns.
[TODO: collect some issues / examples]
(1) #17494
Describe the solution you'd like
Implement a version of Radix Hash Joins:
https://15721.courses.cs.cmu.edu/spring2016/papers/balkesen-icde2013.pdf
Describe alternatives you've considered
No response
Additional context
No response
Activity
Omega359 commentedon Nov 26, 2025
This paper is well worth a read here https://db.in.tum.de/~bandle/papers/bandle-partitionVsNonPartition.pdf
"So unless the radix join is beneficial for other
reasons, e.g., larger than main-memory working sets, we express
reservations that implementing the radix join in a general-purpose
production system justifies the added complexity."
Dandandan commentedon Nov 26, 2025
Thank you!
Omega359 commentedon Nov 26, 2025
For anyone curious how radix joins work I present to you Andy Pavlo (who else): https://youtu.be/pqRV7Mss1Fg?si=WPlilqTfHIC-7UMf&t=1813
Dandandan commentedon Nov 27, 2025
Another data point is that DuckDB and other engines implement radix hash join and seem to have good results doing so, maybe there are newer resources showing this improvement better?
TPC-H joins are relatively easy: most joins are on some integer key column without many duplicate values when the build side is the "correct" one. The results from this paper are from TPC-H only (on which we perform already reasonably good for most queries).
Omega359 commentedon Nov 28, 2025
It's quite possible that newer research has better info, or points to where RHJ would be significantly better than the existing HJ or SMJ! I just wanted to bring your attention to the research that I've read on the subject.
alamb commentedon Dec 1, 2025
@XiangpengHao has some thoughts on this topic I think
Also, see https://crates.io/crates/congee for an implementation of a radix tree in high performance rust
saadtajwar commentedon Jun 30, 2026
Hey @Dandandan, happy to help out with this if it's still open! TY!
alamb commentedon Jul 1, 2026
This is one of those projects where we would probably need to do a prototype and get that working well / with high performance, before spending a lot of time polishing / preparing to swtich the main implementation in DataFusion
saadtajwar commentedon Jul 1, 2026
Hmm I see, thanks @alamb - I'll hold off on this for now :)
duongcongtoai commentedon Jul 16, 2026
I did some personal research myself, i think they are not implementing radix hash join (even though in source code they do refer things like RadixPartition). They are implementing HHJ: https://duckdb.org/library/saving-private-hash-join-vldb/