Repository navigation
Support compression in spill files #16130
Description
Activity
I think this issue can be added to your current TODO list @ding-young.
In the future, if we introduce a dedicated spill format, we might revise the implementation to apply compression to the entire spill file.
But I think it's good to be implemented now since the code footprint is small (should be just passing an option into Arrow IPC reader/writer)Reacted by ding-youngSure. I also think that introducing compression option first is good to start though we may try more complicated techniques.
take
related/prior art: #14078 (comment)
Reacted by ding-young
Is your feature request related to a problem or challenge?
Part of #16065
In memory-limited queries, temporary results will be written to disk to reduce memory pressure. Now those spills are using plain arrow encoding, they can be applied some general-purpose compression like lz4 and zstd for more efficient disk space usage.
Describe the solution you'd like
Describe alternatives you've considered
No response
Additional context
No response