You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Build multi-column null-aware NOT IN on the subquery side (multi-key null-aware RightAnti) #26089
Is your feature request related to a problem or challenge?
A multi-column (a, b) NOT IN (SELECT x, y ...) (#25737, #19857) plans as a null-aware LeftAnti (or LeftMark) hash join, and only a single-key null-aware join is swapped by JoinSelection. So the outer side is always the build side and is held in memory, even when the subquery is small. @comphead showed the effect in #19857 (comment):
CREATETABLEbigASSELECT CASE WHEN v <0 THEN NULL ELSE v END AS a, v % 1000AS b FROM generate_series(1, 2000000) t(v);
CREATETABLEsmallASSELECT CASE WHEN v <0 THEN NULL ELSE v END AS x, v % 1000AS y FROM generate_series(1, 100) t(v);
-- datafusion-cli --memory-limit 64MSELECTcount(*) FROM big WHERE a NOT IN (SELECT x FROM small);
-- RightAnti, builds on `small`: runsSELECTcount(*) FROM big WHERE (a, b) NOT IN (SELECT x, y FROM small);
-- LeftAnti, builds on `big`: Resources exhausted ... Failed to allocate additional 75.6 MB for HashJoinInput
The NULL path also pairs every NULL-valued row with every row on the other side (#26088).
Describe the solution you'd like
A multi-key null-aware RightAnti that builds on the subquery side, deduplicates it, and decides each outer (probe) row as it streams, the same shape as PostgreSQL's findPartialMatch. Each probe row is TRUE unless some build row in its correlation scope matches it exactly or has no definite mismatch with it.
Describe alternatives you've considered
Keeping the outer side as the build side and only narrowing the candidate pairs (#26088). That helps the NULL path but not the memory use.
Additional context
The null_aware_join benchmark suite has Q10 to Q12 for the multi-column shapes.
Activity
viirya commentedon Oct 6, 2026
I'll take this one once #19857 is merged, since it builds on the multi-column null-aware join added there.