Repository navigation
More accurate memory accounting in external sort #14748
Description
Activity
There is a small optimization can be done after we have accurate memory accounting #15017 (comment)
I wonder if this is ready to do now?
I wonder if this is ready to do now?
Yes, I think so. However, this might cause a temporary slowdown due to an extra
row <-> colconversion. Since the performance can be brought back by reusing converted rows between operators, and also I believe correctness is more important, this is not a hard blocker.
I plan to work on this ticket in the future, but if anyone is available to take it now, please feel free.Reacted by Andrew Lambtake
I filed an epic with a bunch of related memory items to try and coordinate some response
Is your feature request related to a problem or challenge?
#14644 fixed an external sorting bug. Each batch's memory overhead is estimated as
2 * batch memory sizefor the extra row conversion overhead. It works for common cases but still can fail. See #14644 (comment) and #14644 (comment).To implement a more accurate memory size estimation for those edge cases, we can first do
col->rowconversion and measure the memory consumption. (depends on #7053)Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response