[rustyconover]

DuckDB 2.0 can spill aggregate states to disk, but only for states that live in the aggregate arena, have no destructor, and register export and import callbacks. Our states were heap objects with destructors. So DuckDB couldn't see their memory, and memory_limit didn't apply to them.

20M rows, 2M groups, 8 threads	before	after
datasketch_kll(200, v), 1 GB limit	39 GB peak, 22 s	0.77 GB, 6.0 s
datasketch_hll(12, v), 4 GB limit	3.3 GB, not counted	2.1 GB, counted
approx_quantile (built-in)	out of memory at 1 GB and 4 GB	—

What changed

• DSAllocator arena mode: it allocates from the arena of the aggregate callback that's currently running (DSArenaScope) and never frees. Each sketch grows only in an arena that the current thread is allowed to use. When states combine, the source is copied into the target's arena rather than taken over.
• Arena states for KLL and HLL: they have no destructor, and export and import their sketches as BLOBs. KLL over VARCHAR stays on the heap, because std::string items allocate their own memory.
• KLL item buffer: DuckDB creates a state per input row when grouping reduces little (MaybeSkipLookups). So KLL states keep their first 32 items in an array that grows as needed, and build a sketch only when more arrive; results are unchanged. Small exact input sketches are buffered the same way, and buffered states export as raw items, so spilled states don't come back full size.
• Error messages: DuckDB's own errors, such as out of memory, now pass through deserialization unchanged instead of being reported as corrupt input.

Library patches (in the vcpkg overlay, port-version 3)

• hll-union-allocator.patch: fix: hll_union kept memory of input sketches' allocators apache/datasketches-cpp#556. Without it, HLL unions kept heap copies of their inputs.
• ebpps-merge-qualified-swap.patch: fix: ebpps_sketch::merge(const&) called an unqualified swap apache/datasketches-cpp#557. This lets us drop the EBPPS copy-then-merge workaround.
• kll-lazy-items.patch: perf: kll_sketch allocates its items on first use apache/datasketches-cpp#558.

Tests

New test/sql/datasketch_spill.test, run under a 128MB limit with 4 threads:

• 400k-group KLL and HLL aggregates
• merging stored sketches
• EXPORT_STATE round trips
• the VARCHAR heap path

All tests pass on EC2: 32 test files, 1953 assertions.

Still to do (next PRs)

• HLL: it still builds a union for each per-row state, so it needs a value buffer like KLL's.
• The other sketch families: they still use heap states.

🤖 Generated with Claude Code