You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Make BUFFER_CAPACITY_AFTER_SORT_ELIMINATION configurable #21417
#21182 introduces elimination of SortExec through sort pushdown. This resulted in slowdowns for some queries because SortExec (which loads all data into memory for sorting) acted as a buffer between SortPreservingMergeExec and the I/O bound scans. This caused I/O fragmentation and slowdowns. We introduced BufferExec to restore this buffering behavior. This new setup uses strictly less memory than SortExec because it doesn't need to buffer the entire dataset in memory, just enough to keep SortPreservingMergeExec constantly fed. BufferExec does support a memory limit for buffering (and will also respect the global memory limits / spill where needed). We hard coded this memory limit to 64MB since that is enough for our benchmark data, however we think it should be made configurable and defaulted to something more like 1Gi or 2Gi (large enough to hold at least 1 maybe 2 row groups for wide data sets). There is little negative to making this number large since if memory limits are reached BufferExec will spill to disk, and no matter how big we make the number memory use will still be lower than it was with SortExec. It may cause higher peak memory use and not make the query any faster, so a sensible default does make sense. 1Gi seems like a reasonable number.
Activity
feat: make sort pushdown BufferExec capacity configurable, default 1GB
feat: make sort pushdown BufferExec capacity configurable, default 1GB (
feat: make sort pushdown BufferExec capacity configurable, default 1GB (
feat: make sort pushdown BufferExec capacity configurable, default 1GB (