Repository navigation
Busy-waiting in SortPreservingMergeStream #16321
Description
Describe the bug
When running a query like select a from annotated_data_infinite2 order by b desc limit 10, a SortPreservingMergeStream is created that merge sorts the presorted partitions. During the initial sorting phase of the query, the query execution will continuously hit
I believe this was introduced in f2a8b07. It's not clear to me why the self wake was added.
To Reproduce
- Add
print!("P"); stdout().flush();just before the return at https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/sorts/merge.rs#L236 - Run a sort on a large dataset with target partitions > 1 so that
SortPreservingMergeStreamis used. - Observe how often the
poll_next_inneris hit
Expected behavior
- No busy-waiting during query execution
Additional context
No response
Activity
pepijnve commentedon Jun 7, 2025
Investigating, but help appreciated.
pepijnve commentedon Jun 7, 2025
A closer look shows I might be completely mistaken. Will close if irrelevant.Nope, not a case of PEBKAC after all. I've refined the reproduction scenario.bug: remove busy-wait while sort is ongoing (apache#16321)
bug: remove busy-wait while sort is ongoing (apache#16321)
bug: remove busy-wait while sort is ongoing (apache#16321)
bug: remove busy-wait while sort is ongoing (apache#16321)
bug: remove busy-wait while sort is ongoing (apache#16321)
pepijnve commentedon Jun 8, 2025
Just to have a hyperlink between the issues: as far as I can tell the behavior was introduced by #12302 to resolve #12300.
bug: remove busy-wait while sort is ongoing (apache#16321)
bug: remove busy-wait while sort is ongoing (#16322)