Repository navigation
SMJ producing different results than HashJoin when doing a semi join #10886
Description
Describe the bug
As a part of #10728 we have added tests for semi join, and on the first merge on main they failed (they always passed). I have ignored that tests for now but that failure suggests a bug . It only happened once in several builds, and on the AMD64 architecture
---- fuzz_cases::join_fuzz::test_semi_join_1k_filtered stdout ----
thread 'fuzz_cases::join_fuzz::test_semi_join_1k_filtered' panicked at datafusion/core/tests/fuzz_cases/join_fuzz.rs:416:13:
assertion `left == right` failed: SortMergeJoinExec and HashJoinExec produced different row counts
left: 100
right: 99
stack backtrace:
0: rust_begin_unwind
see https://github.com/apache/datafusion/actions/runs/9475682375/job/26107451352
To Reproduce
https://github.com/apache/datafusion/actions/runs/9475682375/job/26107451352
Expected behavior
Should pass
Additional context
No response
Activity
comphead commentedon Jun 12, 2024
the behavior is not deterministic, I ran tests on my local and it fails 1 run of 3, checking
eejbyfeldt commentedon Jun 13, 2024
I think this should be reopened or we should open a separate issue. The test flaked here: https://github.com/apache/datafusion/actions/runs/9501960117/job/26188803965?pr=10898 and I was able to make it flake locally as well.
jonahgao commentedon Jun 13, 2024
Reopened it. I encountered the same failure. https://github.com/apache/datafusion/actions/runs/9502397050/job/26190267777?pr=10903
comphead commentedon Jun 13, 2024
Thanks @eejbyfeldt and @jonahgao I'll comment the test now and run more tests on top of it to see where the spontaneous bug is.
comphead commentedon Jun 13, 2024
Filed #10904
Will run the test in the loop to find out where the issue is
comphead commentedon Jun 13, 2024
Found that SMJ left semi sometimes have a duplicated row in the output, I'm trying to make a test case
edmondop commentedon Jun 13, 2024
This is interesting! At least breaking the build was worth it
comphead commentedon Jun 20, 2024
I think I'm able to reproduce it