You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Improve test coverage for spilling (memory-limited) sort/aggregation/sort-merge-join #13431
Is your feature request related to a problem or challenge?
Now DataFusion supports spilling for memory-limited sort/aggregation/sort-merge-join operators. In order to stabilize those features #13123, more tests should be added.
The high-level objectives are:
Memory limit stress tests for related operators: Run queries with large memory consumption (sorting a large relational table for example), and use minimal possible memory limit to let them run successfully
Add integration test to make sure physical memory usage respects to configured memory limits
Add fuzz tests to ensure the correctness of memory-limited queries (spilled and in-memory execution should return equivalent results)
Activity
Make Comet opt-in but with all features enabled by default (apache#557)
test: cover partially-ordered aggregate spilling
minor(test): cover partially ordered aggregate spilling (apache#23947)
minor(test): strengthen sort-merge join spilling coverage (apache#23988)
fix: keep every spilled slice of a sort-merge join inner key group (a…
minor(test): cover partially ordered aggregate spilling (apache#23947)
minor(test): strengthen sort-merge join spilling coverage (apache#23988)
fix: keep every spilled slice of a sort-merge join inner key group (a…
minor(test): strengthen bitwise sort-merge join spill coverage (apach…